You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-10 - 05:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Mon Aug 10, 2026 00:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
275832242168,24sleep10-21swapper/119:09:591
273342238163,24sleep20-21swapper/219:06:472
275062205169,23sleep30-21swapper/319:09:023
274252202165,25sleep00-21swapper/019:07:570
874321400,4sleep22767799cyclictest22:05:142
874321400,4sleep22767799cyclictest22:05:132
27677996864,3cyclictest0-21swapper/221:53:042
27677996663,2cyclictest0-21swapper/220:57:312
27677996663,2cyclictest0-21swapper/220:57:312
67442600,2sleep10-21swapper/120:45:221
212522600,7sleep07-21ksoftirqd/021:20:130
132872520,1sleep213286-21netstat21:00:192
310612450,1sleep10-21swapper/121:40:251
128592440,2sleep339-21rcuc/322:15:013
2767299381,11cyclictest7386-21latency_hist00:30:011
2767299381,11cyclictest7386-21latency_hist00:30:011
36632360,3sleep33666-21pmu-power23:05:233
27672993518,14cyclictest1-21systemd19:30:011
27672993514,18cyclictest16675-21sed19:55:171
2767299351,10cyclictest19281-21awk22:30:021
2768499341,24cyclictest0-21swapper/320:59:413
2768499341,24cyclictest0-21swapper/320:59:413
2767299349,3cyclictest18227-21/usr/sbin/munin20:00:201
2767299346,2cyclictest9855-21crond20:55:001
27672993415,16cyclictest31853-21cut00:10:151
2767299341,2cyclictest0-21swapper/122:48:211
27672993411,3cyclictest30784-21/usr/sbin/munin19:15:131
2767299341,10cyclictest0-21swapper/122:31:251
2766799346,13cyclictest32745-21idleruntime-cro19:20:020
2766799342,24cyclictest2921-21kworker/0:220:44:230
27667993411,11cyclictest29664-21awk21:40:020
2768499331,29cyclictest0-21swapper/322:34:123
2768499330,22cyclictest0-21swapper/319:54:183
2767299334,6cyclictest4810-40dnf19:30:031
2767299333,12cyclictest27441-21crond21:35:021
2767299332,12cyclictest30804-21users20:25:261
2767299332,11cyclictest6658-21diskstats22:00:171
2767299331,4cyclictest13606-21awk19:50:011
2767299331,25cyclictest0-21swapper/121:35:371
27672993312,3cyclictest2207-21users19:20:271
27672993311,3cyclictest32695-21/usr/sbin/munin23:00:151
2767299331,11cyclictest13911-21df_inode23:30:141
27672993310,3cyclictest14490-21/usr/sbin/munin21:05:111
2767299330,6cyclictest0-21swapper/100:07:241
2768499321,23cyclictest0-21swapper/319:12:173
2768499321,16cyclictest278732sleep322:47:363
2768499320,25cyclictest0-21swapper/321:42:483
27672993211,3cyclictest32599-21processes20:30:221
2767299321,12cyclictest17739-21munin-run23:40:011
2767299321,11cyclictest23654-21cat22:39:571
27672993210,4cyclictest8508-21/usr/sbin/munin19:35:271
2767299312,12cyclictest5665-21head21:55:251
27672993113,3cyclictest13469-21fschecks_time22:15:151
2767299311,2cyclictest16459-21diskmemload22:07:181
2767299311,2cyclictest16459-21diskmemload22:07:181
2767299311,24cyclictest0-21swapper/122:43:091
2767299311,23cyclictest0-21swapper/122:52:061
2767299311,12cyclictest12303-21cpu21:00:121
2767299310,10cyclictest0-21swapper/123:10:121
2766799312,14cyclictest13344-21munin-run23:30:010
2768499300,10cyclictest0-21swapper/321:10:163
2767299304,3cyclictest87550irq/128-enp1s0-22:10:211
2767299301,16cyclictest0-21swapper/123:19:091
2767299301,16cyclictest0-21swapper/123:19:091
2767299301,12cyclictest827-21runrttasks20:21:251
2767299301,12cyclictest0-21swapper/121:21:411
2767299301,11cyclictest0-21swapper/121:51:001
2767299301,10cyclictest0-21swapper/123:07:371
27672993010,4cyclictest0-21swapper/123:50:121
2767299300,4cyclictest0-21swapper/120:05:121
2767299300,13cyclictest15324-21kworker/u8:223:40:121
2768499291,11cyclictest0-21swapper/320:04:113
2768499291,11cyclictest0-21swapper/319:42:183
2768499290,9cyclictest0-21swapper/321:18:563
2767299299,3cyclictest3218-21latency_hist20:40:011
2767299292,19cyclictest0-21swapper/100:01:241
2767299291,6cyclictest27869-21/usr/sbin/munin19:10:161
2767299291,5cyclictest16693-21/usr/sbin/munin21:10:251
2767299291,16cyclictest0-21swapper/120:12:311
2767299291,12cyclictest0-21swapper/120:57:441
2767299291,12cyclictest0-21swapper/120:57:441
2767299291,10cyclictest0-21swapper/119:44:331
27672992910,3cyclictest21563-21timerwakeupswit23:45:241
2767299290,1cyclictest0-21swapper/121:26:211
2768499281,9cyclictest0-21swapper/321:23:123
2768499281,21cyclictest0-21swapper/322:35:043
2768499281,13cyclictest0-21swapper/320:50:533
2768499281,12cyclictest0-21swapper/320:14:273
2768499281,12cyclictest0-21swapper/300:09:253
2768499281,10cyclictest0-21swapper/322:59:273
2768499281,10cyclictest0-21swapper/322:04:483
2767299288,2cyclictest15324-21kworker/u8:200:20:321
2767299281,2cyclictest0-21swapper/121:45:331
2767299281,13cyclictest20475-21users21:15:291
27672992811,3cyclictest15444-21df_inode22:20:141
2767299281,12cyclictest12433-21ntp_states23:25:191
27672992810,3cyclictest3746-21expr20:40:141
2768499271,9cyclictest0-21swapper/323:10:253
2768499271,9cyclictest0-21swapper/319:35:083
2768499271,2cyclictest0-21swapper/321:53:383
2768499271,11cyclictest0-21swapper/321:04:583
2768499271,11cyclictest0-21swapper/319:18:483
2768499270,2cyclictest0-21swapper/323:24:353
2768499270,11cyclictest0-21swapper/323:42:253
2767299276,3cyclictest32460-21latency_hist23:00:011
2767299271,6cyclictest0-21swapper/100:35:211
27672992715,3cyclictest0-21swapper/100:30:481
27672992714,2cyclictest0-21swapper/100:16:151
2767299270,15cyclictest0-21swapper/120:15:091
2766799272,9cyclictest26702-21kworker/0:120:39:470
2766799271,9cyclictest23642-21kworker/0:322:53:160
2766799271,16cyclictest8967-21kworker/0:220:00:360
2768499261,9cyclictest0-21swapper/323:30:593
2768499261,9cyclictest0-21swapper/321:27:083
2768499261,20cyclictest0-21swapper/322:08:363
2768499261,20cyclictest0-21swapper/322:08:363
2768499261,12cyclictest0-21swapper/321:39:343
2768499261,10cyclictest0-21swapper/323:25:033
2768499261,10cyclictest0-21swapper/300:22:053
2768499260,2cyclictest0-21swapper/323:38:193
2768499260,21cyclictest0-21swapper/322:25:443
2768499260,20cyclictest0-21swapper/321:33:323
2768499260,11cyclictest0-21swapper/323:54:073
2768499260,10cyclictest0-21swapper/300:39:143
2767299265,3cyclictest87550irq/128-enp1s0-23:55:181
2767299260,2cyclictest0-21swapper/119:52:091
2766799261,9cyclictest8967-21kworker/0:219:46:300
2768499251,9cyclictest16459-21diskmemload21:47:543
2768499251,9cyclictest0-21swapper/320:21:573
2768499251,12cyclictest0-21swapper/320:18:373
2768499251,10cyclictest0-21swapper/320:30:473
2768499250,11cyclictest0-21swapper/323:47:413
2768499250,11cyclictest0-21swapper/320:38:413
2768499250,11cyclictest0-21swapper/300:18:213
2768499250,10cyclictest0-21swapper/322:17:543
2767299251,13cyclictest16459-21diskmemload23:20:321
2766799253,15cyclictest30911-21kworker/0:000:12:410
2766799252,9cyclictest9373-21kworker/0:000:36:450
2766799252,9cyclictest2921-21kworker/0:220:55:070
2766799252,9cyclictest2921-21kworker/0:220:55:070
2766799252,9cyclictest28944-21kworker/0:220:29:000
2766799252,9cyclictest12627-21kworker/0:122:17:020
27667992514,2cyclictest1480-21kworker/0:021:52:270
2768499245,2cyclictest0-21swapper/323:02:093
2768499241,13cyclictest0-21swapper/319:58:093
2768499241,11cyclictest0-21swapper/320:48:173
2768499241,11cyclictest0-21swapper/320:25:513
2768499241,10cyclictest0-21swapper/320:40:593
2768499241,10cyclictest0-21swapper/319:34:043
2768499241,10cyclictest0-21swapper/300:02:473
2768499240,16cyclictest0-21swapper/322:42:483
2768499240,13cyclictest0-21swapper/300:11:113
2766799249,3cyclictest3718-21kworker/0:221:59:110
2766799247,2cyclictest7105-21kworker/0:100:27:070
2766799247,2cyclictest7105-21kworker/0:100:27:070
2766799242,8cyclictest10709-21kworker/0:219:11:580
2766799242,11cyclictest562-21kworker/0:020:34:020
2766799241,9cyclictest17448-21kworker/0:223:47:390
2766799241,11cyclictest32306-21kworker/0:223:03:060
2766799240,9cyclictest11054-21kworker/0:319:44:100
2768499238,3cyclictest0-21swapper/323:57:533
2768499231,7cyclictest0-21swapper/319:20:043
27684992310,2cyclictest0-21swapper/319:28:003
2768499230,9cyclictest0-21swapper/323:18:353
2768499230,9cyclictest0-21swapper/323:18:353
2768499230,9cyclictest0-21swapper/322:20:303
2768499230,9cyclictest0-21swapper/321:59:043
2768499230,12cyclictest0-21swapper/321:06:343
2768499230,11cyclictest0-21swapper/322:54:163
2766799238,2cyclictest21231-21kworker/0:122:35:140
2766799232,8cyclictest6558-21kworker/0:323:14:460
2766799231,8cyclictest8967-21kworker/0:220:05:260
2766799231,8cyclictest5964-21kworker/0:122:04:550
2766799231,8cyclictest31621-21kworker/0:221:43:270
2766799231,8cyclictest22261-21kworker/0:320:14:100
27667992310,3cyclictest4888-21kworker/0:300:34:550
2768499221,9cyclictest0-21swapper/300:30:283
2768499221,7cyclictest0-21swapper/319:48:083
2766799229,3cyclictest8967-21kworker/0:219:50:100
2766799228,3cyclictest8251-21kworker/0:022:11:020
2766799228,2cyclictest8775-21kworker/0:020:51:270
2766799227,2cyclictest4450-21kworker/0:019:33:360
2766799222,9cyclictest11763-21kworker/0:321:01:070
2766799222,8cyclictest14842-21kworker/0:222:26:100
2766799222,8cyclictest14000-21kworker/0:021:05:430
2766799222,6cyclictest27144-21kworker/0:321:33:150
2766799221,9cyclictest30002-21kworker/0:122:57:140
2766799221,9cyclictest21231-21kworker/0:122:33:260
2766799221,8cyclictest15202-21kworker/0:123:39:030
2766799221,5cyclictest0-21swapper/000:09:350
2766799221,11cyclictest32477-21kworker/0:319:22:110
2766799221,11cyclictest29373-21kworker/0:121:46:110
2766799220,9cyclictest6558-21kworker/0:323:20:360
2768499210,10cyclictest0-21swapper/300:25:483
2768499210,10cyclictest0-21swapper/300:25:483
2767799212,11cyclictest0-21swapper/221:30:232
27677992116,3cyclictest14482-21grep22:15:252
27677992114,4cyclictest4444-21df21:55:122
2766799219,2cyclictest32477-21kworker/0:319:29:140
2766799218,2cyclictest5274-21kworker/0:320:46:210
2766799216,9cyclictest4888-21kworker/0:300:24:450
2766799212,9cyclictest18515-21kworker/0:221:18:230
2766799212,9cyclictest17736-21kworker/0:319:59:080
27667992114,4cyclictest9136-21grep22:05:160
27667992114,4cyclictest9136-21grep22:05:160
2766799211,11cyclictest24171-21kworker/0:023:53:350
2766799210,11cyclictest12977-21kworker/0:023:32:260
2768499201,11cyclictest0-21swapper/320:09:053
2767799207,4cyclictest5472expr23:00:132
2767799201,9cyclictest0-21swapper/222:27:172
2767799201,6cyclictest0-21swapper/221:36:532
2767799201,5cyclictest0-21swapper/200:25:222
2767799201,5cyclictest0-21swapper/200:25:222
27677992014,3cyclictest20854-21dump-pmu-power21:20:012
2767799201,11cyclictest0-21swapper/222:12:382
2766799206,2cyclictest24460-21kworker/0:020:15:120
2766799204,2cyclictest19672-21kworker/0:023:44:590
2766799201,8cyclictest14842-21kworker/0:222:21:460
2766799201,11cyclictest2543-21/usr/sbin/munin23:05:250
2767799191,6cyclictest0-21swapper/219:43:502
2767799191,15cyclictest369-21taskset00:11:022
2767799191,12cyclictest0-21swapper/221:10:192
2766799199,3cyclictest23898-21/usr/sbin/munin22:40:180
2766799197,2cyclictest699-21kworker/0:100:15:350
2766799196,3cyclictest24460-21kworker/0:020:22:580
2766799191,8cyclictest0-21swapper/023:57:230
2766799190,8cyclictest23642-21kworker/0:322:46:100
262712190,4sleep226275-21timerwakeupswit20:15:252
46072181,3sleep20-21swapper/219:27:392
2767799182,3cyclictest0-21swapper/222:34:292
2767799181,8cyclictest22076-21needreboot21:20:192
2767799181,7cyclictest29886-21users22:50:262
2767799181,7cyclictest0-21swapper/223:34:092
2767799181,7cyclictest0-21swapper/223:12:042
2767799181,7cyclictest0-21swapper/220:00:352
2767799181,6cyclictest0-21swapper/223:54:542
2767799181,6cyclictest0-21swapper/222:02:052
2767799181,6cyclictest0-21swapper/200:02:292
2767799181,6cyclictest0-21swapper/200:00:012
2767799181,5cyclictest0-21swapper/219:59:192
27677991814,3cyclictest0-21swapper/223:20:372
2767799181,15cyclictest0-21swapper/222:40:152
27677991811,4cyclictest22426-21seq20:10:022
2767799181,10cyclictest0-21swapper/222:45:252
2767799180,10cyclictest0-21swapper/220:41:342
2766799188,2cyclictest16246-21kworker/0:121:11:550
2766799185,2cyclictest20737-21kworker/0:021:26:490
2766799181,8cyclictest24171-21kworker/0:000:01:030
2766799181,4cyclictest7232-21/usr/sbin/munin19:35:150
73942171,2sleep20-21swapper/200:30:012
2767799178,3cyclictest5401-21awk19:30:152
2767799174,3cyclictest87350irq/126-enp1s0-20:30:192
2767799174,3cyclictest32-21ksoftirqd/219:11:182
2767799171,7cyclictest3218-21latency_hist20:40:002
2767799171,6cyclictest28984-21/usr/sbin/munin00:05:192
2767799171,6cyclictest20668-21cut23:45:142
2767799171,6cyclictest0-21swapper/221:49:342
2767799171,6cyclictest0-21swapper/200:17:272
2767799171,5cyclictest24184-21sed21:25:182
2767799171,3cyclictest0-21swapper/221:41:492
2767799171,15cyclictest0-21swapper/200:35:352
2767799171,14cyclictest0-21swapper/223:35:062
2767799171,14cyclictest0-21swapper/220:46:302
2767799171,14cyclictest0-21swapper/200:20:032
2767799171,10cyclictest0-21swapper/222:55:502
27677991710,4cyclictest0-21swapper/220:10:312
2766799175,2cyclictest32256-21kworker/0:023:17:520
2766799175,2cyclictest32256-21kworker/0:023:17:520
235442171,2sleep20-21swapper/222:38:342
133942171,2sleep20-21swapper/219:47:342
2767799165,5cyclictest16382-21switchtime22:20:242
2767799161,9cyclictest0-21swapper/220:23:532
2767799161,6cyclictest11408-21cpu23:25:112
2767799161,5cyclictest0-21swapper/223:41:452
2767799161,5cyclictest0-21swapper/223:20:012
2767799161,5cyclictest0-21swapper/223:20:012
2767799161,5cyclictest0-21swapper/219:50:242
2767799161,3cyclictest0-21swapper/220:26:212
2767799161,3cyclictest0-21swapper/219:36:582
2767799161,2cyclictest0-21swapper/220:50:302
27677991612,3cyclictest0-21swapper/221:08:322
27677991611,3cyclictest3919-21sort23:05:252
27677991611,3cyclictest30738-21acpi19:15:042
27677991611,3cyclictest1763-21grep19:20:222
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional