Reading library
Evaluate whether a search feature helps its reader
Evaluate a quantization search through a small, explicit reading task: observable decisions, honest response limits, controlled walkthroughs and documented unmet needs.
Name the user decision
Start with a decision narrower than find useful Reddit information. In a proposed LocalLLaMA exercise, ask a model developer to find a historical discussion that states the conditions of a quantization memory measurement, or explain why the available material is insufficient. The second outcome matters: an honest unresolved question may be more useful than a confident answer built from missing context. This is a suggested evaluation task, not evidence that a matching discussion exists or that the current feature has helped real users complete it.
Define success in terms the interface can support. Can the reader identify the submitted conditions, inspect the available source text, distinguish their own note from that text, and decide whether to refine or stop? Do not make a complete survey of Reddit or the globally best answer the acceptance criterion. The service is an independent historical archive with bounded retrieval, optional fields and no completeness guarantee. A successful HTTP request is evidence of transport and retrieval, not proof that the reader's research question has been answered.
Choose observable evidence
Record what happens during the task: which filters the reader recognizes, what condition they change, which text supports their decision, and where they hesitate. Distinguish an interface observation from an explanation inferred afterward. If a reader says a title looks promising, ask what body context confirms it rather than counting the click as a correct answer. Records arrive in archive _doc order, not relevance, date or score order. Treat the first displayed result as a reading opportunity, not an automatic recommendation to validate.
Use measures that the available evidence can justify. You can record task steps, requests made and problems encountered in an actual session, but this proposal supplies no benchmark numbers. Precision or recall claims need a defined relevance judgment and an appropriate reference set; a few retrieved pages do not establish archive-wide recall. An exact or lower-bound pagination total also says nothing about answer quality. Timing one request would describe that observation, not a performance guarantee. Keep such limits beside any measurements you later collect.
Run a small task walkthrough
Begin with the proposed request /v1/search?q=quantization&subreddit=LocalLLaMA&limit=3&page=1. The caller's backend supplies the tenant key; the participant sees the question and submitted conditions. Ask them to inspect a returned reference and explain what is present or missing. If memory remains the issue, let them add that one word deliberately. Explain field-local AND: the analyzed terms must match within a matching title or text field. Do not promise exact phrases, typo correction, semantic understanding or a complete conversation that the contract does not provide.
Use controlled local fixtures to examine outcomes an available response may not cover: a missing title, missing context, successful empty data, an HTTP error and a response that never arrives. Label these as synthetic and record any walkthrough actually performed separately from this plan. Check whether the participant can recover the submitted query and choose a next action. For continuation, respect pagination.has_more and the 100-page limit; a full final window is not a completed survey. Further successful requests, including empty ones, each consume one credit.
Record unmet needs
Write findings with a specific observation and its consequence. For example, a future session might reveal that a reader mistook a lower-bound total for an exact number, or could not tell a failed continuation from an empty result. These are examples of reportable findings, not results from a study already run. Separate interface changes you can make from absent evidence you cannot manufacture. Clearer labels may help with a missing date; they cannot recover an unavailable source or certify the conditions of an undocumented experiment.
Keep the evaluation bounded to the participants, tasks and fixtures actually used. A small walkthrough does not produce a ranked list of community problems or a prevalence estimate. Record unresolved needs alongside successful task steps, and identify what evidence a later check would require. Accessibility or device-specific behavior should be reported only after testing it. The decision to continue development should rest on what readers demonstrably understand and can do, while retaining uncertainty about historical coverage, unseen material and conclusions the search service cannot establish.
