Wiki

Appearance

Task creation ​

Each task is defined by a single .toml file. The file should be located inside the web/tasks/ directory.

On this page ​

Task file structure ​

The task file is a .toml file that contains the following fields.

  • [task]

    • name - the name of the task
    • template - path to a .S or .c file, that will be used as a template for the task
    • description - the description of the task, can be styled using markdown, MathJax can also be used
    • c_solution - (optional), if set to true, the task is set to be solved in C, Makefile and file to be present during compile time may be needed.
    • cache_max_size - (optional), if set to some int number, sets the maximum size of the cache in the simulator (activates the cache settings in the evaluator)
    • submit_start and submit_end - (optional), if set to a timestamp in a format of 2024-01-01T00:00:00Z the task will be available for submission only in the given time frame

  • [arguments]

    • run - arguments that will be passed to QtRVSim when evaluation the task, note that some arguments are required for certain functionalities, eg. --d-regs should be used, when reading the state of registers is needed. By default --dump-cycles is the minimal required argument in this section.

  • [[inputs]] array - array of textual inputs descriptions that will be displayed to the user, now are not needed to be filled out

    • data_in - data that will be passed to the task
    • data_out - data that is expected as an output
    • description - description of the input

Simple preprocessor ​

New feature

You can now use a preprocessor to dynamically generate random data for your testcases, mainly to prevent hardcoded solutions.

Start your task file with a preprocessor variables section:

toml
[preprocessor]
vect_10 = "[random.randint(0, 100) for _ in range(10)]"
vect_10_s = "sorted(vect_10)"

Where variables are defined by python functions.

To use the variables in a testcase, surround it by {{ and }}:

toml
[[testcases.starting_mem]]
array_size = [10]
array_start = "{{$vect_10$}}"

[[testcases.reference_mem]]
array_start = "{{$vect_10_s$}}"
Details

The surrounding symbols are technically "{{$ and $}}", this is because non numeric or array like values need to be treated like strings, otherwise they would lead to a not toml-readable file.

This is why the python code for variable assignment and the variables itself are surrounded by "" and are effectively treated as strings.

The most important field is the [[testcases]], which is an array that can have following fields:

toml
[[testcases]]
name = "test01"

Ending reference registers:

toml
[[testcases.reference_regs]]
a1 = 10
a2 = 12

Starting memory ranges can be defined by:

toml
[[testcases.starting_mem]]
0x400 = [5, 10]
0x404 = [10, 15]

The reference (ending / expected) memory ranges:

toml
[[testcases.reference_mem]]
0x408 = [15]

A testcase can be set to private by doing:

toml
[[testcases]]
name = "scoring testcase"
private = true

A scoring testcase (benchmark) can be defined by the follofing:

toml
[score]
description = "Scoring based on the number of cycles used to execute the program."
testcase = "scoring testcase"

INFO

It is best to define the scoring testcase to be a private one, so that the users cannot see the specific scoring testcase that will be run on their code.

Input UART:

toml
[[testcases.input_uart]]
uart = "112233\n445566\n"

Expected output UART:

toml
[[testcases.reference_uart]]
uart = "557799\n"

If your tasks requires to be compiled to elf, before running in the simulator (eg. when %lo and %hi are usedm or C), you need to create a Makefile inside the [make] section:

toml
Makefile="""ARCH=riscv64-unknown-elf
SOURCES = submission.S
TARGET_EXE = submission

...

clean:
	rm -f *.o *.a $(OBJECTS) $(TARGET_EXE) depend

"""

You can also supply any number of files that will be created and used in compile time by:

toml
[[files]]
name = "crt0local.S"
code = """
...
"""
[[files]]
name = "test.S"
code = """
...
"""

INFO

The Makefile should accept a submission.S file at the same folder, and create a submission executable in the same folder. There should also be a clean target, that is called after the evaluation of the task.

Look below at the more complex examples to see exact Makefile usage.

Examples ​

Reading from registers ​

The most basic example of a task file, only uses register values to check the correctness of the solution.

toml
[task]
name = "Simple value addition"
template = "S_templates/addition.S"
submit_start = "2024-01-01T00:00:00Z"
submit_end = "2024-12-31T23:59:59Z"

description = '''
# Simple value addition

Write a program that loads a value `10` into register a1 and value `12` into register a2.
Then, add the values and store the result in register a3.
'''

[arguments]
run = "--d-regs --dump-cycles --cycle-limit 100"

[[inputs]]
data_in = "None"
data_out = "Save the result of the addition in register a2."
description = "Sample task, simple loading of values"

[[testcases]]
name = "test01"

[[testcases.reference_regs]]
a1 = 10
a2 = 12

[[testcases]]
name = "test02"

[[testcases.reference_regs]]
a3 = 22

[score]
description = "Scoring based on the number of cycles used to execute the program."
testcase = "test02"

Reading from memory ​

Also uses memory values to load values to the program and to check the final state of memory addresses. Look at how multiple memory addresses can be set and checked.

toml
[task]
name = "Read an save to memory"
template = "S_templates/memory.S"

description = '''
# Read and save to memory

Write a program that loads 2 32-bit values from memory starting at the adress 0x400 into two registers (a0 and a1).
Then add the values in a0 and a1, and store the result in a2.
Save the result in memory after the two values that were loaded.
'''

[arguments]
run = "--d-regs --dump-cycles --cycle-limit 100"

[[inputs]]
data_in = "Two values in memory starting at address 0x400."
data_out = "The values in registers, the sum in register a2, and the sum in memory after the two values."
description = "Loading of values from memory."

[[testcases]]
name = "test01"

[[testcases.reference_regs]]
a0 = 5
a1 = 10
a2 = 15

[[testcases.starting_mem]]
0x400 = [5, 10]

[[testcases.reference_mem]]
0x408 = [15]

[[testcases]]
name = "scoring testcase"
private = true

[[testcases.starting_mem]]
0x400 = [1711, 1989]

[[testcases.reference_mem]]
0x408 = [3700]

[score]
description = "Runtime of the program in cycles."
testcase = "scoring testcase"

Usage of the preprocessor (Bubble sort) ​

An example of a task that uses the preprocessor to generate random data for the testcases.

toml
[preprocessor]
vect_10 = "[random.randint(0, 100) for _ in range(10)]"
vect_10_s = "sorted(vect_10)"
vect_100_l = "[random.randint(5000, 10000) for _ in range(100)]"
vect_100_l_s = "sorted(vect_100_l)"
vect_23 = "[random.randint(0, 50) for _ in range(23)]"
vect_23_s = "sorted(vect_23)"

[task]
name = "Bubble Sort"
template = "S_templates/bubble.S"

description = '''
# Bubble Sort.
**Write a program that sorts an array using bubble sort algorithm.**

The size of the array will be located at the address `array_size`, the integer array (32-bit integer words) will start
at the address `array_start`.

The program should sort the array in ascending order.
'''

[arguments]
run = "--dump-cycles --cycle-limit 100000"

[[inputs]]
data_in = "Size of the array located at address array_size, the array start is located at the address array_start."
data_out = "Sorted array of length array_size"
description = "Cache optimized sorting."

[[testcases]]
name = "5 elements"

[[testcases.starting_mem]]
array_size = [5]
array_start = [1, 3, 4, 5, 2]

[[testcases.reference_mem]]
array_start = [1, 2, 3, 4, 5]

[[testcases]]
name = "10 elements"

[[testcases.starting_mem]]
array_size = [10]
array_start = "{{$vect_10$}}"

[[testcases.reference_mem]]
array_start = "{{$vect_10_s$}}"

[[testcases]]
name = "100 elements (random vector with large numbers)"

[[testcases.starting_mem]]
array_size = [100]
array_start = "{{$vect_100_l$}}"

[[testcases.reference_mem]]
array_start = "{{$vect_100_l_s$}}"

[[testcases]]
name = "private"
private = true

[[testcases.starting_mem]]
array_size = [23]
array_start = "{{$vect_23$}}"

[[testcases.reference_mem]]
array_start = "{{$vect_23_s$}}"

[[testcases]]
name = "scoring testcase"
private = true

[[testcases.starting_mem]]
array_size = [4]
array_start = [4, 3, 2, 1]

[[testcases.reference_mem]]
array_start = [1, 2, 3, 4]


[score]
description = "Runtime of the program in cycles."
testcase = "scoring testcase"

Pragma cache usage ​

An example of a usage of the #pragma cache directive in the task template file.

toml
[preprocessor]
vect = "[random.randint(0, 100) for _ in range(100)]"
vect_s = "sorted(vect_100)"

[task]
name = "Cache optimization"
template = "S_templates/cache.S"
cache_max_size = 16

description = '''
# Cache optimization.
**Write a program that sorts an array using a sorting algorithm of your choice.**

The size of the array will be located at the address `array_size`, the integer array (32-bit integer words) will start
at the address `array_start`.

The program should sort the array in ascending order.
Your program will be tested with different array sizes and different values in the array.

Set the parameters of the cache by using `#pragma cache:policy,sets,words_in_blocks,ways,write_method` on a separate line in your task submission.
For example: `#pragma cache:lru,1,1,1,wb`

This parameter will be passed to qtrvsim_cli as the `--d-cache` parameter.

- The allowed maximal cache capacity is limited to 16 32-bit words.
- The length of the official evaluation dataset to sort is in the range of 24 to 32 words.
- The initial main memory memory access latency is set to 10 cycles.
- The burst latency 2 is configured for the following consecutive accesses.

The complete task description can also be found [here](https://cw.fel.cvut.cz/wiki/courses/b35apo/en/homeworks/bonus/start), Makefile and template files for your own testing can be found on [GitLab](https://gitlab.fel.cvut.cz/b35apo/stud-support/-/tree/master/seminaries/qtrvsim/apo-sort).
'''

[arguments]
run = "--dump-cycles --cycle-limit 10000 --read-time 10 --write-time 10 --burst-time 2"

[[inputs]]
data_in = "Size of the array located at address array_size, the array start is located at the address array_start."
data_out = "Sorted array of length array_size"
description = "Bubble sort algorithm."

[[testcases]]
name = "100 elements"

[[testcases.starting_mem]]
array_size = [100]
array_start = "{{$vect$}}"

[[testcases.reference_mem]]
array_start = "{{$vect_s$}}"

[[testcases]]
name = "private 1"
private = true

[[testcases.starting_mem]]
array_size = [10]
array_start = [10, 9, 8, 7, 6, 5, 4, 3, 2, 1]

[[testcases.reference_mem]]
array_start = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

[[testcases]]
name = "scoring testcase"
private = true

[[testcases.starting_mem]]
array_size = [4]
array_start = [4, 3, 2, 1]

[[testcases.reference_mem]]
array_start = [1, 2, 3, 4]

[score]
description = "Runtime of the program in cycles."
testcase = "scoring testcase"

Usage of MathJax and syntax highlighting (Fibonacci) ​

Here you can see the usage of formatted task file with MathJax and syntax highlighting.

toml
[task]
name = "Fibonacci sequence without the hazard unit"
template = "S_templates/fib.S"

description = '''
Write a code for the calculation of the N-th Fibonacci number (for \\(N > 2\\)). The Fibonacci sequence is defined as follows:

$$ F(n) = F(n-1) + F(n-2) $$ for \\( n > 2 \\), and $$ F(0) = 0, F(1) = 1 $$

Here are the first few numbers in the Fibonacci sequence:
$$ 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144,... $$

Modify the code in such a way that it does have no hazards (the code produces correct results in the pipelined execution without the hazard unit). Minimize the number of nops in your code. Prefer instruction reordering whenever possible.

The input number 'n' is at location 'input'. Save your result to location output. Execute with pipeline with no hazard unit.

## A possible solution in C
    :::c
    t0 = 5;  //  Set value of N
    s0 = 0;  //  F(0)
    s1 = 1;  //  F(1)

    for(t1 = 2; t1 <= t0; t1++)
    {
            t2 = s0 + s1;
            s0 = s1;
            s1 = t2;
    }

'''

[arguments]
run = "--dump-cycles --pipelined --hazard-unit none --cycle-limit 1000"

[[inputs]]
data_in = "Size of the array located at address fibo_limit."
data_out = "Array of Fibonacci numbers starting at the address fibo_series."
description = "Data hazard prevention."

[[testcases]]
name = "3"
[[testcases.starting_mem]]
input = [3]
[[testcases.reference_mem]]
output = [2]

[[testcases]]
name = "4"
[[testcases.starting_mem]]
input = [4]
[[testcases.reference_mem]]
output = [3]

[[testcases]]
name = "5"
[[testcases.starting_mem]]
input = [5]
[[testcases.reference_mem]]
output = [5]

[[testcases]]
name = "8"
[[testcases.starting_mem]]
input = [8]
[[testcases.reference_mem]]
output = [21]

[[testcases]]
name = "15"
[[testcases.starting_mem]]
input = [15]
[[testcases.reference_mem]]
output = [610]

[[testcases]]
name = "larger"
private = true
[[testcases.starting_mem]]
input = [17]
[[testcases.reference_mem]]
output = [1597]

[[testcases]]
name = "benchmark"
private = true
[[testcases.starting_mem]]
input = [40]
[[testcases.reference_mem]]
output = [102334155]

[score]
description = "Runtime of the program in cycles."
testcase = "benchmark"

Usage of custom Makefile (Hazard detection) ​

This task uses a custom Makefile to compile the solution to an ELF file, which is then run in the simulator (for programs that cannot yet be run in the simulator directly).

toml
[task]
name = "Data hazard prevention"
template = "S_templates/hazards.S"

description = '''
# Data hazard prevention

**Implement a Fibonacci series computation algorithm for pipelined processors without a hazard unit.**

Program will load the length of the Fibonacci series from `fibo_limit`, and should save it into
an array of 32-bit integer words starting at the address `fibo_series`.

The complete task description can also be found [here](https://cw.fel.cvut.cz/wiki/courses/b35apo/en/homeworks/bonus/start), Makefile and template files for your own testing can be found on [GitLab](https://gitlab.fel.cvut.cz/b35apo/stud-support/-/tree/master/seminaries/qtrvsim/fibo-hazards).
'''

[arguments]
run = "--dump-cycles --pipelined --hazard-unit none --cycle-limit 5000"

[[inputs]]
data_in = "Size of the array located at address fibo_limit."
data_out = "Array of Fibonacci numbers starting at the address fibo_series."
description = "Data hazard prevention."

[[testcases]]
name = "5 Fibonacci numbers"

[[testcases.starting_mem]]
fibo_limit = [5]

[[testcases.reference_mem]]
fibo_series = [0, 1, 1, 2, 3]

[[testcases]]
name = "10 Fibonacci numbers"

[[testcases.starting_mem]]
fibo_limit = [10]

[[testcases.reference_mem]]
fibo_series = [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]

[[testcases]]
name = "scoring testcase"
private = true

[[testcases.starting_mem]]
fibo_limit = [25]

[[testcases.reference_mem]]
fibo_series = [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597, 2584, 4181, 6765, 10946, 17711, 28657, 46368]

[score]
description = "Runtime of the program in cycles."
testcase = "scoring testcase"

[make]
Makefile="""ARCH=riscv64-unknown-elf

SOURCES = submission.S
TARGET_EXE = submission

CC=$(ARCH)-gcc
CXX=$(ARCH)-g++
AS=$(ARCH)-as
LD=$(ARCH)-ld
OBJCOPY=$(ARCH)-objcopy

ARCHFLAGS += -mabi=ilp32
ARCHFLAGS += -march=rv32i
ARCHFLAGS += -fno-lto

CFLAGS  += -ggdb -Os -Wall
CXXFLAGS+= -ggdb -Os -Wall
AFLAGS  += -ggdb
LDFLAGS += -ggdb
LDFLAGS += -nostartfiles
LDFLAGS += -nostdlib
LDFLAGS += -static
#LDFLAGS += -specs=/opt/musl/riscv64-linux-gnu/lib/musl-gcc.specs

CFLAGS  += $(ARCHFLAGS)
CXXFLAGS+= $(ARCHFLAGS)
AFLAGS  += $(ARCHFLAGS)
LDFLAGS += $(ARCHFLAGS)

OBJECTS += $(filter %.o,$(SOURCES:%.S=%.o))
OBJECTS += $(filter %.o,$(SOURCES:%.c=%.o))
OBJECTS += $(filter %.o,$(SOURCES:%.cpp=%.o))

all : default

.PHONY : default clean dep all run_test

%.o:%.S
	$(CC) -D__ASSEMBLY__ $(AFLAGS) -c $< -o $@

%.o:%.c
	$(CC) $(CFLAGS) $(CPPFLAGS) -c $< -o $@

%.o:%.cpp
	$(CXX) $(CXXFLAGS) $(CPPFLAGS) -c $<

%.s:%.c
	$(CC) $(CFLAGS) $(CPPFLAGS) -S $< -o $@

default : submission

$(TARGET_EXE) : $(OBJECTS)
	$(CC) $(LDFLAGS) $^ -o $@

dep: depend

depend: $(SOURCES) $(glob *.h)
	echo '# autogenerated dependencies' > depend
ifneq ($(filter %.S,$(SOURCES)),)
	$(CC)  -D__ASSEMBLY__ $(AFLAGS) -w -E -M $(filter %.S,$(SOURCES)) \
	  >> depend
endif
ifneq ($(filter %.c,$(SOURCES)),)
	$(CC) $(CFLAGS) $(CPPFLAGS) -w -E -M $(filter %.c,$(SOURCES)) \
	  >> depend
endif
ifneq ($(filter %.cpp,$(SOURCES)),)
	$(CXX) $(CXXFLAGS) $(CPPFLAGS) -w -E -M $(filter %.cpp,$(SOURCES)) \
	  >> depend
endif

clean:
	rm -f *.o *.a $(OBJECTS) $(TARGET_EXE) depend

-include depend

"""

Writing the solution in C, custom Makefile, files and UART input/output (Calculator) ​

This example of a task file requires the user to solve the task in C, write the output to UART. The task also uses other files which are present during compile time.

toml
[task]
name = "Simple Calculator"
template = "S_templates/calculator.c"
c_solution = true

description = '''
# Simple Calculator

**Implement a program, that loads two numbers from the serial port,
adds them together and prints them to the serial port.**

The C language program execution requires at least a minimal startup code sequence,
even for the simplest bare-metal environment.
The C code ABI for RISC-V requires at least to set global pointer (`gp`) register,
see the file [crt0local.S](https://gitlab.fel.cvut.cz/b35apo/stud-support/-/blob/master/seminaries/qtrvsim/uart-calc-add/crt0local.S).

The file `crt0local.S` will be included in the execution directory and linked with your program.

The complete task description can also be found [here](https://cw.fel.cvut.cz/wiki/courses/b35apo/en/homeworks/bonus/start), Makefile and template files for your own testing can be found on [GitLab](https://gitlab.fel.cvut.cz/b35apo/stud-support/-/blob/master/seminaries/qtrvsim/uart-calc-add).
'''

[arguments]
run = "--dump-cycles --cycle-limit 5000"

[[inputs]]
data_in = "Two numbers present at the serial port."
data_out = "Their sum to the serial port."
description = "Simple calculator."

[[testcases]]
name = "test1"

[[testcases.input_uart]]
uart = "111\n222\n"

[[testcases.reference_uart]]
uart = "333\n"

[[testcases]]
name = "test2"

[[testcases.input_uart]]
uart = "72235\n3254777\n"

[[testcases.reference_uart]]
uart = "3327012\n"

[[testcases]]
name = "scoring testcase"
private = true

[[testcases.input_uart]]
uart = "112233\n445566\n"

[[testcases.reference_uart]]
uart = "557799\n"

[score]
description = "Runtime of the program in cycles."
testcase = "scoring testcase"

[make]
Makefile="""ARCH=riscv64-unknown-elf
#ARCH=riscv64-linux-gnu

SOURCES = submission.c crt0local.S
TARGET_EXE = submission

CC=$(ARCH)-gcc
CXX=$(ARCH)-g++
AS=$(ARCH)-as
LD=$(ARCH)-ld
OBJCOPY=$(ARCH)-objcopy

ARCHFLAGS += -mabi=ilp32
ARCHFLAGS += -march=rv32i
ARCHFLAGS += -fno-lto

CFLAGS  += -ggdb -Os -Wall
CXXFLAGS+= -ggdb -Os -Wall
AFLAGS  += -ggdb
LDFLAGS += -ggdb
LDFLAGS += -nostartfiles
LDFLAGS += -nostdlib
LDFLAGS += -static
#LDFLAGS += -specs=/opt/musl/riscv64-linux-gnu/lib/musl-gcc.specs

LOADLIBES += -lgcc

CFLAGS  += $(ARCHFLAGS)
CXXFLAGS+= $(ARCHFLAGS)
AFLAGS  += $(ARCHFLAGS)
LDFLAGS += $(ARCHFLAGS)

OBJECTS += $(filter %.o,$(SOURCES:%.S=%.o))
OBJECTS += $(filter %.o,$(SOURCES:%.c=%.o))
OBJECTS += $(filter %.o,$(SOURCES:%.cpp=%.o))

all : default

.PHONY : default clean dep all run_test

%.o:%.S
	$(CC) -D__ASSEMBLY__ $(AFLAGS) -c $< -o $@

%.o:%.c
	$(CC) $(CFLAGS) $(CPPFLAGS) -c $< -o $@

%.o:%.cpp
	$(CXX) $(CXXFLAGS) $(CPPFLAGS) -c $<

%.s:%.c
	$(CC) $(CFLAGS) $(CPPFLAGS) -S $< -o $@

default : submission

$(TARGET_EXE) : $(OBJECTS)
	$(CC) $(LDFLAGS) $^ $(LOADLIBES) -o $@

dep: depend

depend: $(SOURCES) $(glob *.h)
	echo '# autogenerated dependencies' > depend
ifneq ($(filter %.S,$(SOURCES)),)
	$(CC)  -D__ASSEMBLY__ $(AFLAGS) -w -E -M $(filter %.S,$(SOURCES)) \
	  >> depend
endif
ifneq ($(filter %.c,$(SOURCES)),)
	$(CC) $(CFLAGS) $(CPPFLAGS) -w -E -M $(filter %.c,$(SOURCES)) \
	  >> depend
endif
ifneq ($(filter %.cpp,$(SOURCES)),)
	$(CXX) $(CXXFLAGS) $(CPPFLAGS) -w -E -M $(filter %.cpp,$(SOURCES)) \
	  >> depend
endif

clean:
	rm -f *.o *.a $(OBJECTS) $(TARGET_EXE) depend

-include depend

"""

[[files]]
name = "crt0local.S"
code = """/* minimal replacement of crt0.o which is else provided by C library */

.globl main
.globl _start
.globl __start

.option norelax

.text

__start:
_start:
	.option push
	.option norelax
	la gp, __global_pointer$
	.option pop
	la      sp, __stack_end
	addi    a0, zero, 0
	addi    a1, zero, 0
	jal     main
quit:
	addi    a0, zero, 0
	addi    a7, zero, 93  /* SYS_exit */
	ecall

loop:	ebreak
        beq     zero, zero, loop

.bss

__stack_start:
	.skip   4096
__stack_end:

.end _start

"""