Campus AI
A bilingual retrieval assistant for Ain Shams Faculty of Science
Computer Science graduation project · Grade A− · Live
The problem
Faculty of Science students lose a lot of time to questions that already have documented answers — course registration rules, academic warning thresholds, faculty contacts, GPA calculation. The information exists, spread across PDFs and portal pages, mostly in Arabic. Nobody reads it.
A chatbot is the obvious answer. It is also the easy part.
The system
A Flutter application over a Python retrieval backend. Documents are chunked and embedded into ChromaDB; queries run through hybrid retrieval — dense semantic search combined with BM25 keyword matching — with optional cross-encoder reranking. The model generates over the retrieved context, streamed to the app over SSE, with multi-turn conversation memory and query rewriting for follow-up questions.
It runs against a local model, and against a hosted API. That was not the original plan — see below.
How I measured it
The interesting question was never whether it produced fluent Arabic. It was whether the retrieved context was actually the right context.
I built two evaluation layers:
- A 182-case test set covering the question types students actually ask, scored for whether the system returned the correct information.
- A 21-question Arabic gold-standard set, hand-labelled, scored on RAGAS dimensions — faithfulness, relevancy, and completeness.
The two measure different things. The first asks did it get the answer right. The second asks was the answer grounded in what it retrieved, and was it complete.
Results
The first run scored 54.4% on the 182-case set. Failure analysis showed the errors were not random — they clustered around specific preprocessing problems: inconsistent Arabic normalisation, chunk boundaries splitting related information, and retrieval returning topically adjacent but wrong passages.
Table view
| Stage | Accuracy |
|---|---|
| first run | 54.4% |
| after preprocessing fixes | 97% |
| Change | +42.6 pts |
The second layer scores the generated answer itself, on a hand-labelled Arabic set.
Table view
| Dimension | Score |
|---|---|
| Faithfulness | 0.85 |
| Relevancy | 0.79 |
| Completeness | 0.54 |
| Overall | 0.73 |
Limitations
Completeness at 0.54 is the weakest number here, and it is the honest one. The system reliably retrieves correct information and stays faithful to it, but often returns a partial answer where the full one spans multiple source documents. Fixing that means multi-document synthesis, not better retrieval — a different problem, and the one I would work on next.
The evaluation sets are also mine, not an external benchmark. They reflect the questions I anticipated.
On running locally and on an API
The system was built for local inference — the point was that student data never leaves the machine. Close to the defense, with limited time and no guarantee about network conditions in the room, I added a hosted API path so the demo could not stall mid-answer.
Rather than replace one with the other, I kept both. The system now runs either way. It was a deadline decision that turned into a better architecture.
What I would do differently
Document the evaluation in the repository, not just the report. The numbers are the most defensible part of this project and they currently live outside the code.