Interface ScreenAPI

Remarks

Coordinates use the same signed physical-pixel space as Mouse, Window, and overlays. Monitors left of or above the primary monitor have negative coordinates. Throws a runtime error when Windows cannot read the pixel, including coordinates outside the virtual screen.

Remarks

Returns a two-element tuple: an ARRAY of every distinct place that cleared the threshold, strongest first, and ALWAYS a diagnostics object. Destructure it: const [hits, why] = await Screen.FindImage(path). CHECK hits.length, never hits itself, because an empty array is truthy in JavaScript: if (hits) runs when nothing was found and hits[0].x then throws. Write if (hits.length) and take hits[0] as the best match. WHY AN ARRAY RATHER THAN ONE MATCH: a screen genuinely holds zero, one, or several copies of a thing. Two identical buttons used to return one arbitrary winner, decided by nothing the caller could see or predict, and acting on that is a coin flip. hits.length > 1 is the signal that a region, or a more distinctive crop, is needed before clicking anything. Entries are distinct positions, never one element reported twice. DIAGNOSTICS ARE FOR DEBUGGING, AND NOTHING PRINTS THEM FOR YOU: they are a return value, not a log line. A search that fails tells you nothing from the outside, because the one visible symptom, an empty hits, is the same for every cause. The moment something is wrong, print them and read the numbers rather than guessing: Console.Log(JSON.stringify(why, null, 2)). Read them once when you write the search, too, not only when it breaks. A high score is not by itself evidence the search is safe, and the two limits that produce a confident WRONG click, a near-uniform crop and a crop whose detail is all on its boundary, are visible only in why.templateNearUniform and why.templateContrast on a call that reported success. A loop that has checked those once may then drop to const [hits] = ... and ignore the rest. Do this before changing any setting. Roughly half of getting a search working is finding the right threshold for the crop you have, and the other half is narrowing region so a polling loop does not over-tax the CPU, and both of those are measurements whose numbers are in why. FINDING THE RIGHT threshold FOR A CROP: do not guess it, and do not leave it at the default because the default sounds safe. Print why.score twice: once with the element visible on screen, once with it absent. The first is what this crop actually achieves at its true location; the second is what this screen scores without it. Set threshold between the two. If the two numbers are close together there is no value that separates the states, and the crop is the problem rather than the setting: recapture it with more of the element's own detail, or with less changing background baked in. Note the score you measured somewhere, because it is the baseline that later tells you whether a change helped. HOW TO GUIDE SOMEONE TO A WORKING SEARCH: do not guess at settings. Run the search once and read why, which measures the template, the search area and the outcome, then change one thing. why.reason names the outcome and why.advice is a sentence you can show the user as-is. BRANCH ON why.reason, NOT ON THE SCORES: the verdict is decided inside the search, against measured floors, and reason is where it comes out:

• matched: at least one place cleared the threshold; every entry in hits carries a centre. hits.length > 1 means the screen holds several copies and you must disambiguate, by region or by a more distinctive crop, before clicking anything.
• notOnScreen: nothing in the AREA THAT WAS SEARCHED resembles the template. If a region was passed, suspect the region FIRST: it is a fixed rectangle and the element may simply have moved out of it, so drop the region and search the whole screen once before touching the template. With no region, the template is from another screen or another resolution, or the element is not visible yet; confirm it is actually on screen before changing the crop.
• templateTooLarge: the template does not fit inside the area searched, so it could not have matched anywhere. Either a whole screenshot is being used as a template, or region is smaller than the thing being looked for. Widen the region or crop smaller; nothing about the template's appearance is relevant.
• colorMismatch: the shape was found and its colour disagreed. Recapture from the current theme, or pass matchColor: false to confirm the diagnosis.
• brightnessMismatch: a night-light, HDR or theme change.
• contrastMismatch: the element is dimmed or disabled.
• captureUnavailable: the screen came back blank, so nothing in this object is about the template. Windows returns an empty image instead of an error when it cannot read the screen: a game running EXCLUSIVE fullscreen rather than borderless windowed, a window that excludes itself from capture, or a locked session. capturedContrast is the measurement. This outranks every other reason, because when it fires none of the others can be true. Do not look for a setting that fixes exclusive fullscreen: it bypasses the Windows compositor, so no capture API can read it, and PowerKeys overlays and pickers cannot draw over it either. Tell the user to run the game borderless windowed, which on Windows 10 and later performs the same.
• plainTemplate: the crop carries so little structure that it is scored by absolute colour difference, which matches any area of a similar colour. No setting fixes this; the user must crop something with internal detail. Do NOT derive that verdict yourself from why.score against why.structureScore. Both are maxima taken independently over the whole searched area, so they need not describe the same position, and a high structure score is not on its own evidence the element was present: absent templates still reach 0.42 through 0.84 structurally. The search only reports a colour family when the shape score clears a floor well above the default threshold, precisely so that a 0.84 on an absent element is not explained as a colour problem. Reading the two numbers as a story sends users to recapture templates and change display settings over elements that were never on screen. Use them as magnitudes, and reason as the answer. WHAT YOU CAN MEASURE: why reports templateWidth/templateHeight, templateContrast (under about 5 is nearly flat), templateCoverage (below 0.6 takes a slower, less reliable path), elapsedMs, and searchedWidth/searchedHeight. Use those to give concrete advice instead of general advice. WHAT YOU CAN CHANGE, roughly in order of how often it is the answer: the template itself (crop, size, opacity), region, threshold, then sizeTolerance / rotationTolerance / matchColor. Two limits worth knowing before spending turns on them. sizeTolerance stops at 0.25, which covers 125% display scaling and does NOT reach 150%, the Windows default on a 4K monitor; past that the only fix is a template captured at the size it will be searched at. And matchColor: false is not free: over twelve unrelated template/screen pairs the colour-verified default finds nothing, while shape-only matching falsely finds three, at 0.81 through 0.84. All three were the same mostly transparent template, so read this as colour verification carrying most of the safety for a sparse crop rather than as a flat rate; an opaque crop has more of its own margin. Prefer recapturing the template, and reserve matchColor: false for confirming a diagnosis rather than shipping it. A notOnScreen reading is evidence about the TEMPLATE only if the element was actually visible during that run. For anything of the form "click it when it appears", the element is absent on most polls by design, so ask the user to bring it on screen and run once more before suggesting they change the crop. MAKING IT FAST: cost tracks the area searched, so region is the setting that decides what a polling loop costs. Measure it rather than assuming: print why.elapsedMs next to why.searchedWidth/why.searchedHeight and compare that against how often the loop runs. A search that takes longer than the sleep between polls means the loop never rests, which is the case that pegs a core. Once a search succeeds, why.suggestedRegion is a rectangle around where the element actually appeared, sized so it can drift by its own width or height and still be found. Pass it as region on later calls. It is present only when the call matched AND no region was passed in, so capture it from the first whole-screen hit and keep it: once you are passing a region, no further suggestions arrive. A good pattern is to search the whole screen once, keep suggestedRegion, and poll inside it after that, printing elapsedMs once to confirm the drop. Scores: a pixel-faithful template scores 0.9 or higher at its true location, while absent or unrelated content stays below the 0.8 default. Do not read that as a 0.7 ceiling: a mostly transparent template measured 0.78 against an unrelated screen, so the margin is thin for those and threshold is worth raising. A template captured from a different rendering of a live scene (for example a tooltip over changing 3D content baked into the crop) scores materially lower; either lower threshold for that template or crop it from the same visual state it must match. Erasing the changing backdrop to transparency also works and is supported, but expect a lower score rather than a higher one: masking away a background removes the contrast between the element and its surroundings, which is most of what the match is measuring. How much lower depends on the template, so read the score your own template returns and set threshold from that rather than assuming a band. A mostly transparent template is also slower to search, because too few masked pixels remain for the search to shrink the image far. A template throws when its visible area is under 16, measured as the SUM OF ALPHA rather than a count of pixels: a 100-pixel template at 10% opacity sums to 10 and is rejected, so faded templates fail this earlier than their pixel count suggests. A near-uniform template matches by absolute color difference because correlation is undefined without texture. Unknown fields and out-of-range values in options throw. A search rejects rather than resolves when the script is stopped mid-scan, so a polling loop that must clean up should wrap the call in try/catch; the rejection is expected on stop and is not a fault in the template. Both tolerances are ignored for a near-uniform template, which is only ever scored at its nominal pose. imagePath values are resolved canonically. Paths inside this script's native-owned workspace are accepted, and .. escapes are denied. Absolute paths outside the workspace are accepted only when the path was selected by the user through UI.AddFilePicker or is inside a directory selected through UI.AddDirectoryPicker. That grant is scoped to this script, persists only on the computer where the picker was used, and survives local restarts; synced picker values never grant another installation or script. Use FindImageBase64 when embedding a template directly in a script. A file picker returns an empty string until the user actually picks something, and an empty path resolves to the asset directory itself, which fails to decode with an unhelpful message before any diagnosis exists. Guard the first run: check template.GetValue() is non-empty and ask the user to pick a file if it is not, rather than letting the search throw. Polling and cost: a search scans the whole area you give it, so in a loop the region is the setting that decides how much CPU the script uses. Pass the smallest rectangle the target can appear in. Note that a maximized or fullscreen game's Window.Find() bounds cover the entire monitor, so passing window bounds does not narrow anything for a fullscreen target: give explicit coordinates around where the element actually appears, for example the lower third of the screen for an action prompt. Searching every monitor at full size a few times a second is the expensive case, and region is what fixes it. Template size is NOT a matter of cropping as tightly as possible. Cost is not monotonic in template area: measured over a whole 4K desktop, a 64x87 icon costs about 30 ms, a 160x160 crop about 19 ms, a 390x268 dialog about 35 ms, and a 780x536 crop about 94 ms. A very small template is more expensive than a moderate one, because the coarser the target the more places on screen resemble it and the more of them the search has to rank. Somewhere around 150x150 to 400x400 is the cheapest range; crop to the element, not to a fragment of it. TWO TEMPLATE SHAPES COST TWENTY TO EIGHTY TIMES THAT, and neither is exotic. The search shrinks the screen before scanning it, and it can only do that while the template survives shrinking. It stops when the template holds fewer than about 500 visible (alpha-weighted) pixels, or when its SMALLER SIDE is under about 13 pixels, and then every position is scanned at full resolution. Measured on the same 4K desktop against that 30 ms icon: the same icon with three quarters of its pixels erased takes 787 ms, a 400x13 strip 1599 ms, a 12x400 strip 2720 ms, and a 390x268 dialog erased down to scattered glyphs 2694 ms. A polling loop sleeping 500 ms between calls never rests at those costs. templateCoverage and templateWidth/templateHeight are how you see it coming, and elapsedMs is how you confirm it: a bar, divider or progress strip needs padding out to at least ~16 px on its short side, and an erased template needs more of itself kept. THREE LIMITS THAT PRODUCE A CONFIDENT WRONG CLICK, all measured, none visible in any score. A high score is not by itself evidence the search is safe, so check the template against these before trusting a polling loop. First, a crop needs internal DETAIL, and templateContrast cannot see the difference. Sweeping 48 to 120 pixel crops from one screen against three unrelated ones, the worst false match scored 0.9987 from a crop reporting templateContrast 50.1, ten times the value the docs call flat: a straight boundary between two flat areas has a large spread and almost no information. Gradients and elements with generous empty margins behave the same way. How many such crops a given screen yields depends entirely on where you cut it, so treat the mechanism as the finding and not the count. Crop something with glyphs, an icon or texture inside it, and treat a crop that is mostly flat regions as unusable however high its contrast reads. Second, the search matches APPEARANCE, not STATE: an element that differs only in a small part of the crop still matches. Measured on one real element whose correct state scores 1.000, dimming the whole thing to 40% brightness still scored 0.998, a cooldown sweep darkening its lower quarter 0.917, and replacing a fifth of it outright 0.956, every one of them reported as matched. "Click it only once it is enabled" therefore cannot be expressed by threshold alone: the right and wrong states here sit 0.002 to 0.044 apart, while unrelated content is 0.3 or more below both, so any bar that excludes the wrong state also excludes most of the right one. Crop so the part that CHANGES between the states is a large fraction of the template, and verify by measuring both states as above; if they score within roughly 0.1 of each other, no threshold will separate them. Third, SHAPE and SIZE decide how safe a crop is, and templateContrast cannot see either. A crop that is wide and short is the dangerous one, because screens are full of horizontal bars, gradients and text baselines for it to agree with. Measured against two screens the template was absent from: a 400x10 strip scored 0.89 and reported eleven separate hits, 400x13 scored 0.89, and 400x18 scored 0.89, all of them above the 0.8 default and all on visibly different content, while the SAME content cropped as 400x32 scored 0.84 and the same strip turned on its side, 10x400, scored 0.60. Small crops behave the same way: 12x12 through 21x21 crops of ordinary textured UI scored 0.86 to 0.97 against unrelated screens, and stopped doing it at 32x32. So a health bar, a cooldown strip, a divider or a tiny glyph needs either a taller crop that includes something around it, a region that already excludes the lookalikes, or a threshold measured as above rather than left at the default. hits.length > 1 is the live warning that this is happening. Template advice: a fully opaque crop is both the fastest and the most reliable template. Masking most of a template to transparency, leaving only glyphs or a thin outline, gives the search less to lock onto. It scores more readily against unrelated content of a similar color and brightness, and it is not marginally slower but several times slower: a mostly transparent template cannot use the deepest search levels at all. Measured like for like on a 4K desktop, a 31%-opaque crop cost 249 ms against 31 ms for an opaque crop of the same dimensions, which is 8x, and erasing further crosses the cliff described above rather than continuing gently. Prefer an opaque crop, and raise threshold when a template must be mostly transparent. DPI contract: The captured screen and template are compared as physical image pixels. The template's decoded pixel dimensions are used as-is; PowerKeys does not scale it for monitor DPI. Capture the template at the same rendered scale as the target, or set options.sizeTolerance for small size differences. Returned x/y are the match center, not its top-left corner. The decoded template and captured search area are each limited to 67,108,864 pixels.

Remarks

Returns [hits, diagnostics], the same pair Screen.FindImage returns. hits is an ARRAY of every distinct place that cleared the threshold, strongest first: check hits.length and read hits[0], never if (hits), because an empty array is truthy. The second element is always present and explains the outcome through reason, advice, structureScore and suggestedRegion. It is a return value and nothing prints it, so Console.Log(JSON.stringify(why, null, 2)) when a search misbehaves; a script that already works can ignore it. See Screen.FindImage for how to read it when guiding someone to a search that works, including how to measure a threshold for a crop and how to narrow region so a loop stays cheap. Branch on reason rather than comparing score with structureScore yourself; the same reasons apply here, templateTooLarge included. Scores: a pixel-faithful template scores 0.9 or higher at its true location. Do not assume a fixed ceiling for absent content: a mostly transparent template measured 0.78 against an unrelated screen, so measure threshold for your own template as Screen.FindImage describes rather than picking a remembered band. Invalid Base64 or undecodable image data throws, a template whose visible area sums to less than 16 alpha-weighted pixels throws, and unknown fields or out-of-range values in options throw. Use case: Embed template images directly in the script without relying on external files. Encode the image to Base64 once and store as a constant. The string must be plain Base64 and nothing else: a data:image/png;base64, prefix and any embedded newlines from wrapping a long literal are both rejected, which surfaces only as Base64 decode failed. Any image format the decoder recognizes works, not only PNG, and transparency is optional; an opaque crop is still the faster and more reliable choice. DPI contract: The template's decoded pixel dimensions are compared directly with physical screen pixels; no monitor-DPI scaling is applied. Capture the template at the same rendered scale as the target, or set options.sizeTolerance for small size differences. Returned x/y are the match center in signed physical virtual-screen coordinates. The decoded template and captured search area are each limited to 67,108,864 pixels.