Annotation Tips
Tips from the Most Experienced Annotators
Naomi suggests:
Always go to GONG H-Alpha Viewer when unsure
Listen to music as you work
Have designated times for working
Do not rush, it will just yield sloppy results which you will have to fix later
Aya suggests:
Increase the contrast of the image to identify and annotate filaments much better
Make sure of the filaments' chirality from GONG H-Alpha Viewer
When not confident or feel confused always seek help from someone who knows better
When annotating the filament zoom in so that you can include threads much better
Focus on the quality of the annotations, not the quantity
Learn from the feedback comments
Save specific time in each day only for annotating
Riley suggests:
How to draw precise segmentations:
Learn all the shortcuts in the annotation platform.
Use a comfortable mouse.
Use both the auto-annotate and the brush tools.
Use the brush size accordingly, but I mostly use them under or around 10.
Get comfortable with the chirality. I have to look for the texture and details for some of them.
You can use GONG H-Alpha Viewer when appropriate.
How to find time"
I usually draw the segmentations between working, during breaks (since it doesn’t require intense attention), or when I want to feel productive but don’t want a lot of pressure.
Practice more, and you’ll get faster!
A Good Segmentation
A good segmentation holds the following features:
it does not spill over the actual filament,
it does not miss any part of the filament,
it does not cover background noise as part of the filament,
(for Left and Right filaments) it captures the orientation of the threads, and
it does not capture details which are not backed up by the actual filament.
A Good Spine
A good segmentation holds the following features:
it starts from one very end to the other very end,
throughout the path, it stays in the middle of the filament,
it does not go outside the segmentation area, especially around sharp turns, and
it is made of enough joints so that it does not have sharp edges.
Unqualified Images
When an image (or part of it) is blurry and unclear, annotators should use GONG H-Alpha Viewer, (see tutorial). When an image is corrupted (such as those shown on the right), it should be labeled as "Not Qualified" without any annotation made.
This shows an example of an image being marked (incorrectly) as "Not Qualified" because it was blurry.
Please recall that a blurry image is not unqualified. Here are some examples of unqualified images.
Not Enough Information?
A good example (Left: annotated by an annotator, Right: Actual Image):
This might be a reoccurring mistake:
To know whether the big filament is actually one giant filament or a few lined-up filaments, there is no way except looking at its evolution (GONG H-Alpha Viewer is your friend.) See this tutorial.
The big filament should be identified as "Right", not "Ambiguous". The key is to look at the textural patterns (the NE-to-SW tiny threads), instead of looking at the big barbs.
Using "auto-annotate" and "brush" Tool
The example shows a segmentation which does not capture the barbs of the filament.
Annotators can get a very good segmentation by only using the auto-annotate tool. This can be done by only a few clicks.
That said, they must use the brush tool to capture the barbs they already identified.
What a Rejected Annotation Looks Like
The reviewers try to leave comments when they reject an image. Annotators should make sure they read them, and provide corrections accordingly.
An image may be rejected for one or more reasons. When you receive an image back, you only need to fix the issues. The rest of your work will remain there.
Read the comment in this example! These mistakes are frequent ones.
Segmentation Should Capture the Chirality
When a filament is identified as "Left" or "Right", the annotator's segmentation should capture the reasons that made them assign that label. Those reasons are the tiny threads and their orientation.
The example on the right shows a filament (left figure), a good segmentation (middle), and a very good segmentation (right) which captures the orientation of the threads.
Here is another filament (left image). While the annotator has done a good job in capturing the big barbs (middle image), the orientation of threads (smaller barbs) is not captured. In the modified version (right image), those threads are captured as well. (Full size image)
A Few Examples of Good Annotation
Note how the orientation of threads are captured in the segmentation.
How to Speed Up Annotation
A few tips for speeding up without sacrificing the quality:
Use a mouse instead of a touch-pad. Touch-pads will burn you out quickly.
Use short-keys (e.g., v for moving around, f for adding with the brush, e for erasing with the eraser, h for hiding/unhiding your annotations).
Using auto-annotate tool you can make a very nice segmentation before you start using the brush tool. You only need to learn how exactly the inside/outside clicks expand/shrink the selection area.
Not using auto-annotate tool can also be a good idea for some filaments. You can start by a the brush tool if you have a good idea of the shape of the filament. You can then refine it with a finer brush.
Do not use polygon tool. Our objects have much more complicated shapes than what a simple polygon can afford. This is exactly why the Brush tool exists.
When using the brush tool, avoid too-small brush sizes (1 is too small!). It slows you down significantly.
Annotate iteratively! You can go over a batch of 10-20 images quickly and come back and work on more difficult filaments (those which may need you to use the H-Alpha Viewer). This iterative approach will help a lot. Just don't forget to click on "Sent to Review" at the end.
Another big help is a simple fact: the less rejected images you get, the fewers images you will annotate in total. By not being meticulous enough, you may go over 2X-4X images. Being meticulous really pays off.
Lastly, if something is taking to much time for you, ask why (on Slack).