You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-07 - 03:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot7s.osadl.org (updated Wed Jan 07, 2026 00:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
270329066,8sleep20-21swapper/219:09:562
245629065,9sleep30-21swapper/319:06:443
253528963,8sleep00-21swapper/019:07:460
258628764,18sleep10-21swapper/119:08:271
279799220,22cyclictest56550irq/124-enp0s3122:24:582
279099221,20cyclictest0-21swapper/000:05:140
2795992119,1cyclictest0-21swapper/120:45:131
279099211,19cyclictest0-21swapper/022:50:190
279799201,19cyclictest0-21swapper/200:02:032
2790992019,1cyclictest0-21swapper/022:05:130
279799190,3cyclictest0-21swapper/221:05:142
2797991817,0cyclictest0-21swapper/219:45:122
280099170,13cyclictest0-21swapper/320:30:143
279799175,12cyclictest0-21swapper/221:20:502
279799170,1cyclictest0-21swapper/220:50:122
2800991611,2cyclictest0-21swapper/319:15:133
2795991614,1cyclictest0-21swapper/100:35:001
279099163,6cyclictest26696-21perf23:19:590
2800991511,3cyclictest22041-21grep20:55:173
280099143,8cyclictest3848-21run-parts23:40:003
2800991410,2cyclictest26340-21sed19:55:263
280099140,12cyclictest8967-21grep23:50:123
279799142,3cyclictest22146-21acpi23:10:112
2797991414,0cyclictest0-21swapper/220:29:522
2797991412,1cyclictest91932sleep220:30:142
279799141,13cyclictest56550irq/124-enp0s3100:33:152
2797991410,3cyclictest13672-21sh20:40:002
279599140,2cyclictest141rcu_preempt23:50:001
279099144,7cyclictest30985-21apt-get00:35:130
279099141,6cyclictest13362-21munin-run00:00:000
2790991410,2cyclictest31210-21wc20:05:270
2790991410,2cyclictest24723-21cut19:55:140
280099139,3cyclictest6038-21cat21:30:013
280099139,3cyclictest29037-21grep22:15:173
280099139,3cyclictest25032-21head22:05:263
280099139,3cyclictest11569-21df21:40:163
280099139,2cyclictest20788-21wc23:05:183
2800991310,2cyclictest7090-21grep22:35:243
2800991310,2cyclictest4709-21grep19:10:273
2800991310,2cyclictest18158-21grep23:00:173
280099130,12cyclictest8432-21sed20:25:273
279799134,1cyclictest0-21swapper/221:40:152
279799133,9cyclictest4147-21cron20:20:002
279799133,9cyclictest0-21swapper/219:10:132
2797991310,2cyclictest9885-21wc21:35:202
2797991310,2cyclictest9851-21wc23:50:172
2797991310,2cyclictest6220-21sh23:44:592
2797991310,2cyclictest3884-21grep22:30:172
2797991310,2cyclictest30481-21awk22:20:012
2797991310,2cyclictest27848-21head20:00:162
2797991310,2cyclictest20558-21sed00:10:252
279799130,12cyclictest18204-21apache200:06:052
279799130,12cyclictest0-21swapper/219:25:152
279599139,3cyclictest30899-21wc21:14:591
279599139,3cyclictest20221-21grep21:55:281
279599139,3cyclictest20221-21grep21:55:271
279599137,2cyclictest225-21systemd-journal20:10:011
279599133,5cyclictest225-21systemd-journal19:15:001
2795991310,2cyclictest5194-21tail20:20:171
2795991310,2cyclictest30614-21grep23:25:221
279599130,12cyclictest5231-21cat21:25:261
279599130,12cyclictest22038-21head20:55:161
279099138,3cyclictest27989-21tail21:05:270
2790991310,2cyclictest2901-21grep21:20:270
2790991310,2cyclictest25566-21tail21:00:280
2790991310,2cyclictest19227-21grep19:40:270
2790991310,2cyclictest19147-21tr20:50:130
2790991310,2cyclictest15265-21wc00:00:260
280099129,2cyclictest31429-21cut22:20:163
280099129,2cyclictest30789-21grep20:05:213
280099129,2cyclictest23885-21sed19:50:263
280099129,2cyclictest23848-21tail23:10:263
280099129,2cyclictest19187-21tail19:40:273
280099129,2cyclictest1912-21cat20:15:013
280099129,2cyclictest17250-21dump-pmu-power19:40:003
280099129,2cyclictest15418-21wc00:00:263
280099129,2cyclictest11344-21grep22:45:193
280099128,3cyclictest29423-21dpkg20:05:003
280099128,3cyclictest23349-21cron00:20:013
280099128,3cyclictest12274-21cron19:30:003
280099128,2cyclictest3360-21tr22:30:153
280099124,4cyclictest225-21systemd-journal22:55:003
2800991210,1cyclictest6548-21sh20:25:013
279799129,2cyclictest8786-21cat23:50:012
279799129,2cyclictest6073-21sort22:35:142
279799129,2cyclictest4673-21tr20:20:152
279799129,2cyclictest32701-21grep20:10:172
279799129,2cyclictest31521-21sort21:15:152
279799129,2cyclictest18413-21cat23:00:192
279799129,2cyclictest18290-21head19:40:172
279799129,2cyclictest18083-21cron21:55:002
279799129,2cyclictest17309-21wc23:00:002
279799129,2cyclictest17238-21cron19:40:002
279799129,2cyclictest1204-21df22:25:162
279799128,3cyclictest28116-21awk22:15:002
279799124,8cyclictest0-21swapper/223:06:552
279799123,8cyclictest0-21swapper/220:45:112
279799121,4cyclictest25447-21hddtemp_smartct19:55:172
279799121,10cyclictest21289-21apache_volume20:55:122
279599129,2cyclictest7901-21grep20:25:221
279599129,2cyclictest3824-21cron23:40:001
279599129,2cyclictest31208-21grep00:35:151
279599129,2cyclictest28306-21sed20:00:211
279599129,2cyclictest27373-21cat22:10:251
279599129,2cyclictest16844-21grep19:35:251
279599129,2cyclictest11138-21cat19:25:231
279599128,3cyclictest23093-21basename22:05:011
279599128,2cyclictest24398-21grep19:55:001
279599125,6cyclictest26659-21cron23:19:591
279599120,11cyclictest9474-21grep22:40:251
279099129,2cyclictest7451-21wc23:45:180
279099129,2cyclictest3315-21grep19:10:140
279099129,2cyclictest29156-21grep00:30:170
279099129,2cyclictest16826-21grep20:45:140
279099129,2cyclictest16444-21grep19:35:210
279099129,2cyclictest13178-21grep20:35:260
279099128,3cyclictest8480-21sort19:20:180
279099128,3cyclictest7060-21sed19:15:260
279099128,3cyclictest29392-21wc22:15:190
279099128,2cyclictest459-21grep21:15:260
279099128,2cyclictest15495-21wc20:40:260
279099127,3cyclictest542-21wc22:25:010
279099120,11cyclictest0-21swapper/000:25:110
280099118,2cyclictest9332-21grep21:35:173
280099118,2cyclictest7175-21uname21:30:193
280099118,2cyclictest6802-21cat23:45:153
280099118,2cyclictest6281-21cat23:45:003
280099118,2cyclictest32673-21grep00:35:283
280099118,2cyclictest3003-21sed21:20:273
280099118,2cyclictest29382-21grep00:30:183
280099118,2cyclictest27979-21tail21:05:263
280099118,2cyclictest2728-21grep20:15:173
280099118,2cyclictest2634-21chrt22:27:393
280099118,2cyclictest25640-21grep23:15:193
280099118,2cyclictest25220-21wc21:00:243
280099118,2cyclictest23701-21basename00:20:143
280099118,2cyclictest20583-21taskset21:58:573
280099118,2cyclictest20583-21taskset21:58:563
280099118,2cyclictest17646-21cut20:45:193
280099118,2cyclictest16988-21cut00:05:163
280099118,2cyclictest14698-21taskset19:32:543
280099118,2cyclictest12596-21which23:55:233
280099117,3cyclictest8792-21wc19:20:203
279799119,1cyclictest4052sleep223:30:202
279799118,2cyclictest647-21cron00:39:592
279799118,2cyclictest29931-21ls21:10:222
279799118,2cyclictest29663-21tr23:25:152
279799118,2cyclictest27576-21grep23:20:182
279799118,2cyclictest25690-21cut22:10:012
279799118,2cyclictest24398-21sed00:20:172
279799118,2cyclictest22246-21chrt19:50:112
279799118,2cyclictest21355-21ls22:00:172
279799118,2cyclictest13835-21tr21:45:122
279799118,2cyclictest13393-21sed22:50:172
279799118,2cyclictest12342-21cron22:50:002
279799117,3cyclictest14210-21grep20:40:132
279799113,7cyclictest23896-21apache_volume21:00:132
279799112,8cyclictest56550irq/124-enp0s3119:16:452
279799110,10cyclictest0-21swapper/223:18:302
279799110,10cyclictest0-21swapper/220:05:152
279599119,1cyclictest10573-21head23:50:241
279599118,2cyclictest9222-21fschecks_count21:35:151
279599118,2cyclictest7587-21ls22:39:591
279599118,2cyclictest5862-21expr19:15:151
279599118,2cyclictest3856-21seq20:16:241
279599118,2cyclictest32128-21expr23:30:141
279599118,2cyclictest27049-21sed21:05:171
279599118,2cyclictest27003-21head00:25:171
279599118,2cyclictest24245-21uniq00:20:171
279599118,2cyclictest24117-21grep21:00:151
279599118,2cyclictest23793-21grep23:10:261
279599118,2cyclictest21970-21taskset19:49:381
279599118,2cyclictest21016-21ls23:05:211
279599118,2cyclictest20972-21cron00:15:001
279599118,2cyclictest20066-21ntpq20:50:201
279599118,2cyclictest18207-21head23:00:171
279599118,2cyclictest14278-21grep22:50:251
279599118,2cyclictest11082-21cut21:40:151
279599117,3cyclictest6286-21grep21:30:141
279599117,3cyclictest10997-21grep22:45:171
279599117,2cyclictest27022-21grep23:20:151
279599116,2cyclictest452-21systemd-logind00:06:041
279099118,2cyclictest9118-21tr23:50:130
279099118,2cyclictest8006-21grep22:40:130
279099118,2cyclictest7559-21grep20:25:200
279099118,2cyclictest6956-21grep21:30:170
279099118,2cyclictest5800-21readlink20:20:230
279099118,2cyclictest4225-21grep23:40:150
279099118,2cyclictest31993-21grep23:30:140
279099118,2cyclictest30730-21grep23:25:220
279099118,2cyclictest2440-21tail23:35:170
279099118,2cyclictest21566-21cut22:00:180
279099118,2cyclictest21225-21acpi00:15:130
279099118,2cyclictest18805-21grep21:55:150
279099118,2cyclictest18805-21grep21:55:150
279099118,2cyclictest16835-21taskset22:56:140
279099118,2cyclictest15964-21cat21:50:110
279099118,2cyclictest14793-21taskset19:34:090
279099118,2cyclictest11130-21taskset20:33:020
279099117,3cyclictest13-21ksoftirqd/000:10:240
280099108,1cyclictest519-21awk21:15:263
280099108,1cyclictest30667-21taskset21:13:083
280099108,1cyclictest289422sleep323:22:523
280099108,1cyclictest178832sleep321:52:293
280099108,1cyclictest15323-21expr22:55:163
280099108,1cyclictest1398-21taskset23:34:313
280099107,2cyclictest27625-21python00:25:243
280099107,2cyclictest26252-21cat22:10:153
280099107,2cyclictest22005-21cron19:49:593
280099107,2cyclictest18908-21apache_accesses20:50:113
280099107,2cyclictest14863-21wc21:45:193
280099107,2cyclictest14204-21grep20:40:133
280099107,2cyclictest13372-21chrt20:36:243
280099106,2cyclictest21996-21wc22:00:213
279799108,2cyclictest12962-21grep19:30:142
279799108,2cyclictest12827-21sed23:55:262
279799108,1cyclictest7873-21grep19:20:142
279799108,1cyclictest6320-21wget21:30:142
279799108,1cyclictest28080-21taskset00:26:492
279799108,1cyclictest23089-21taskset00:16:452
279799108,1cyclictest1958-21expr23:35:152
279799107,2cyclictest7937-21wget22:40:132
279799107,2cyclictest4141-21expr21:25:152
279799100,9cyclictest18963-21expr21:55:152
279799100,9cyclictest18963-21expr21:55:142
279599108,1cyclictest50652sleep122:33:181
279599108,1cyclictest30728-21seq22:20:101
279599108,1cyclictest25358-21tr19:55:171
279599108,1cyclictest25200-21taskset22:06:561
279599108,1cyclictest22700-21cat00:15:261
279599108,1cyclictest17612-21cut19:40:141
279599108,1cyclictest15558-21taskset21:48:061
279599108,1cyclictest15548-21sed22:55:171
279599108,1cyclictest14269-21expr00:00:141
279599107,2cyclictest32015-21tail21:15:171
279599107,2cyclictest29720-21which22:15:221
279599107,2cyclictest14162-21cat20:40:141
279599107,2cyclictest11291-21idleruntime-cro20:35:001
279599107,2cyclictest11208-21chrt23:55:101
279599107,1cyclictest0-21swapper/120:10:141
279599106,3cyclictest13403-21taskset20:36:451
279599100,9cyclictest97982sleep119:23:401
279599100,9cyclictest1386-21wget21:20:151
279599100,9cyclictest12983-21sort19:30:151
279599100,9cyclictest0-21swapper/123:40:141
279099108,1cyclictest4384-21grep21:25:170
279099108,1cyclictest10395-21chrt21:35:460
279099108,1cyclictest1034-21expr22:25:140
279099107,2cyclictest5433-21taskset22:35:040
279099107,2cyclictest5144-21taskset22:34:200
279099107,2cyclictest32728-21sort20:10:170
279099107,2cyclictest3027-21wc20:15:190
279099107,2cyclictest28410-21swap23:20:260
279099107,2cyclictest27622-21head22:10:260
279099107,2cyclictest27567-21sed20:00:160
279099107,2cyclictest23942-21chrt23:10:430
279099107,2cyclictest21129-21which23:05:210
279099107,2cyclictest20126-21grep19:45:140
279099107,2cyclictest18184-21cut23:00:180
279099107,2cyclictest15175-21wc21:45:240
279099107,2cyclictest12730-21grep21:40:270
279099107,2cyclictest11782-21taskset19:25:300
279099107,1cyclictest10936-21tune2fs22:45:170
279099106,2cyclictest24960-21fgrep00:20:230
279099100,9cyclictest22145-21if_enp2s020:55:170
28009997,1cyclictest206212sleep300:10:323
28009996,2cyclictest8338-21expr22:40:143
28009990,3cyclictest30536-21/usr/sbin/munin23:25:203
27959991,3cyclictest2357-21unixbench_multi22:25:261
27909997,1cyclictest29778-21cat21:10:190
27909997,1cyclictest22739-21cat19:50:140
27959986,1cyclictest16345-21expr21:50:141
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional