I was working on a story on cropping today and then this tweet on cropping popped up on Twitter:

我今天正在撰写一个有关裁剪的故事，然后在Twitter上弹出有关裁剪的推文：

For years and years I was professionally burdened by the cropping of images. Every social media channel or website template design requires some form of cropping. To make images fit, mostly for previewing purposes or as thumbnails. A tedious and humble task, something you would love to see automated. Personally I admire the talent and ingenuity that gave rise to this technology but the huge impact of unintended mistakes and biases asks for more transparency on how these AI’s work. And I was shocked when I repeated the experiment from Bascule and did some others. I had no idea why it happened and I wanted to research.

多年来，我一直因裁剪图像而感到专业负担。 每个社交媒体渠道或网站模板设计都需要某种形式的裁剪。 使图像合适，主要用于预览或用作缩略图。 一个繁琐而谦虚的任务，您希望看到自动化的东西。 我个人很欣赏这种技术的才能和独创性，但是意外错误和偏见的巨大影响要求这些AI的工作方式更加透明。 当我重复从Bascule进行的实验并进行其他操作时，我感到震惊。 我不知道为什么会这样，所以我想研究一下。

手动裁剪 (Manually cropping)

When you need a couple of these sizing/ratio adjustments you do the cropping by hand. Squeezing your images in a much smaller area, especially with different ratio’s between height and width is ugly or not relevant so you take out the digital scissors, make some esthetic considerations, maybe scale the image and cut something of from the sides of the image to make it fit.

当您需要进行一些调整大小/比率的调整时，可以手动进行裁剪。 在较小的区域中挤压图像，尤其是在高度和宽度之间的比率不同时，这很难看或不相关，因此您拿出数字剪刀，进行一些美学考虑，也许缩放图像并从图像的侧面切出一些东西，使它适合。

输入自动控制(Automated cropping with controlled input)

Here you start cropping automatically and you make sure the images going in will turn out OK once some simple cropping procedures are done. The rules might dictate you always focus on the center content and so that requires some caution. Agree some rules on the visuals with yourself and the team and you will be fine. Stupid things will happen but in general you can correct.

在这里，您将自动开始修剪，并确保一旦完成一些简单的修剪过程，进入的图像就会变成OK。 这些规则可能会指示您始终将重点放在中心内容上，因此需要谨慎。 与您自己和团队商定视觉上的一些规则，您会没事的。 愚蠢的事情会发生，但总的来说您可以纠正。

使用AI裁剪用户的照片 (Cropping photo’s from users with AI)

But once you let a user crowd input freely the rules will be broken and you need some different approach to make the cropping work. Here Artificial Intelligence comes in. You set the basic rules to find out the area of interest for cropping and then put the algorithm to (machine) learn on big piles of data and probably assisted by a group of people. Once the code reaches a level of maturity and you are confident it will perform you can start using it on new images that come in from real users. And for instance have it pick a face.

但是，一旦让用户自由输入，规则就会被打破，您需要一些不同的方法来使裁剪工作。 此处出现了人工智能。您设置基本规则以找出感兴趣的种植区域，然后将该算法用于(机器)在大量数据上的机器学习，并且可能需要一群人的帮助。 一旦代码达到了成熟水平，并且您确信它会执行，就可以开始在实际用户输入的新图像上使用它。 例如，让它选择一张脸。

Twitter的故事 (Twitters story)

So how does Twitter approach automatic cropping? Do they look for faces? The only people who know what is in the AI recipe are the cooks. And the nice thing is that Twitter is in this case pretty transparant on how things work and allowed the engineers Lucas Theis and Zehan Wang responsible for automatic cropping to publicly share their story in detail:

那么Twitter如何实现自动裁剪？ 他们在寻找面Kong吗？ 唯一知道AI食谱中内容的人是厨师。 好在这种情况下，Twitter很漂亮透明的关于事情如何运作并允许工程师Lucas Theis和Zehan Wang负责自动裁剪以公开分享他们的故事的详细信息：

In their story they make very clear their Cropping AI is not working with faces. At least not any more.

在他们的故事中，他们非常清楚自己的“裁剪AI”无法与面Kong配合使用。 至少没有了。

Previously, we used face detection to focus the view on the most prominent face we could find. While this is not an unreasonable heuristic, the approach has obvious limitations since not all images contain faces.

以前，我们使用面部检测将视图聚焦在我们可以找到的最突出的面部上。 尽管这不是一种不合理的启发式方法，但由于并非所有图像都包含面部，因此该方法存在明显的局限性。

So when it is not faces that drive the decision, what is then?

因此，当不是面对决定的驱动力时，那又是什么呢？

主题突出 (Saliency of the topic)

The Twitter people explain that a better way to crop is to focus on “salient” image regions. A region having high saliency means that a person is likely to look at it when freely viewing the image. The engineers refer to a research that is used for saliency of a topic.

Twitter人士解释说，一种更好的裁剪方法是专注于“突出”的图像区域。 显着性高的区域意味着人在自由观看图像时可能会看它。 工程师是指一个研究是用于主题的显着性。

We suppose that the most visually salient areas of a photo are also the most relevant ones to the users.

我们假设照片的视觉上最显着的区域也是与用户最相关的区域。

This suggests that the reason for picking one of the faces in the uploaded portraits on Twitter is because the area of the picture is seen as more salient. Saliency is an abstract term but it means: that what jumps out. And although rationally the concept is hard to explain, intuitively you probably sense what is going on. That is because we humans work with saliency as well. But real people would probably not agree on what is the most striking or important part of a picture.

这表明在Twitter上选择上传的肖像中的一张脸的原因是因为图片的区域被认为更加突出。 显着性是一个抽象术语，但它的意思是：突飞猛进。 尽管从理论上讲这个概念很难解释，但直觉上您可能会感觉到发生了什么。 那是因为我们人类也非常重视工作。 但是真实的人可能不会同意图片中最引人注目的或最重要的部分。

The real answer is that “saliency” is based on a combination of visual elements (colour, shape, contrast) and what the people involved in the training of the algorithm (sets of photos) deemed striking/important. The algorithm later applies this saliency concept on newly uploaded photos.

真正的答案是，“显着性”是基于视觉元素(颜色，形状，对比度)以及算法训练中涉及的人员(照片集)的醒目/重要因素的组合。 该算法随后将此显着性概念应用于新上传的照片。

从这往哪儿走 (Where to go from here)

In the Twitter case it is not clear where and how these images were sourced and what made up the jury of people that helped the algorithm to be trained. Twitter already offered to Open Source the cropping code but this maybe calls more for Explainable AI (XAI). This is a procedure where the user can trace back how inferences are made and this sheds light on the black box that we have now. Probably a bit heavy for cropping images on Twitter but very much required for decisions in Education, Insurance, Health Care, etc.

在Twitter案中，尚不清楚这些图像的来源和来源，以及由哪些人组成的陪审团帮助对该算法进行了训练。 Twitter已经向开源提供了裁剪代码，但这可能要求Explainable AI(XAI)更多。 这是一个过程，在此过程中，用户可以追溯进行推理的方式，这使我们了解了现在的黑匣子。 在Twitter上裁剪图像可能有点沉重，但在教育，保险，医疗保健等方面的决策非常必要。

And to prevent mishaps to your own twitter pics I would recommend using the proper sized image. And to speed things up you can use this tool by Sprout:

为了防止您自己的Twitter图片发生意外，我建议您使用适当大小的图像。 为了加快速度，您可以使用Sprout的以下工具：