You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-07 - 08:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Wed Jan 07, 2026 00:44:26)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
151952230166,52sleep30-21swapper/319:07:343
150742203167,24sleep10-21swapper/119:06:001
151692202165,25sleep20-21swapper/219:07:132
150982201165,24sleep00-21swapper/019:06:190
254921580,5sleep01545999cyclictest21:05:140
2136321540,4sleep21547299cyclictest00:15:132
2933021420,4sleep31547999cyclictest00:31:283
164572520,4sleep21547299cyclictest19:10:182
28062500,2sleep3401ktimersoftd/322:20:013
214102500,4sleep01545999cyclictest21:45:390
228272490,3sleep232-21ksoftirqd/221:50:182
4762460,4sleep230-21rcuc/223:25:262
15472994541,3cyclictest0-21swapper/200:05:072
27872400,1sleep10-21swapper/119:50:231
15459993910,13cyclictest20182-21munin-run23:00:010
1547299362,26cyclictest23315-21kworker/2:023:21:062
189872350,4sleep11546699cyclictest22:55:181
1545999338,12cyclictest28995-21awk23:20:000
1546699321,17cyclictest0-21swapper/119:45:181
1545999326,12cyclictest26801-21perf23:15:010
1547299317,3cyclictest19101-21kworker/2:019:35:032
1547299311,20cyclictest19101-21kworker/2:019:30:172
1545999311,14cyclictest3556-21munin-run19:55:010
15459993110,10cyclictest0-21swapper/020:20:000
53602290,4sleep21547299cyclictest22:25:142
1547299299,11cyclictest23315-21kworker/2:021:25:432
1547299292,11cyclictest23315-21kworker/2:000:35:112
15472992917,2cyclictest23315-21kworker/2:000:21:452
15459992910,16cyclictest24942-21idleruntime-cro19:30:020
15472992818,2cyclictest23315-21kworker/2:021:18:532
1547999271,10cyclictest0-21swapper/320:42:383
1547299279,2cyclictest3766-21kworker/2:220:24:502
1547299278,2cyclictest19101-21kworker/2:019:24:242
1547299277,3cyclictest23315-21kworker/2:021:33:192
15472992717,2cyclictest23315-21kworker/2:022:07:062
15472992711,3cyclictest16071-21hddtemp_smartct21:35:152
1547299271,12cyclictest23315-21kworker/2:022:53:142
1545999274,11cyclictest1003-21awk23:30:000
1547299267,2cyclictest23315-21kworker/2:020:46:402
1547299262,13cyclictest23315-21kworker/2:023:41:542
1547299262,12cyclictest23315-21kworker/2:023:32:562
15472992616,2cyclictest23315-21kworker/2:021:21:412
15472992613,2cyclictest3766-21kworker/2:220:26:262
1547299261,18cyclictest23315-21kworker/2:022:13:342
15472992611,13cyclictest23315-21kworker/2:000:10:212
1546699261,23cyclictest0-21swapper/100:00:121
1545999261,4cyclictest0-21swapper/023:21:390
1545999261,12cyclictest2595-21munin-run22:20:010
1547999252,11cyclictest9415-21/usr/sbin/munin22:35:243
1547299252,10cyclictest23315-21kworker/2:023:39:082
1547299252,10cyclictest23315-21kworker/2:023:10:462
1547299251,14cyclictest23315-21kworker/2:022:55:522
15472992510,3cyclictest3766-21kworker/2:220:20:022
1546699252,12cyclictest14201-21munin-run00:00:011
1547999241,15cyclictest0-21swapper/320:02:083
1547999241,11cyclictest0-21swapper/319:17:043
1547999241,10cyclictest0-21swapper/319:47:083
1547299249,3cyclictest3766-21kworker/2:219:55:552
1547299249,3cyclictest19101-21kworker/2:019:46:352
1547299249,2cyclictest23315-21kworker/2:020:54:362
1547299248,9cyclictest23315-21kworker/2:000:00:392
1547299248,2cyclictest23315-21kworker/2:000:28:072
1547299247,4cyclictest1049-21ntp_states21:00:242
1547299242,9cyclictest23315-21kworker/2:021:05:072
1547299242,10cyclictest23315-21kworker/2:023:16:162
15472992411,3cyclictest19101-21kworker/2:019:43:392
15472992411,2cyclictest3766-21kworker/2:220:09:482
1547299241,11cyclictest23315-21kworker/2:021:42:512
15472992410,3cyclictest19101-21kworker/2:019:16:442
1546699242,4cyclictest18640-21munin-run00:10:011
1546699241,13cyclictest8187-21diskstats23:45:141
1545999241,6cyclictest0-21swapper/023:57:010
15479992310,3cyclictest0-21swapper/320:48:183
1547299239,3cyclictest23315-21kworker/2:023:07:322
1547299239,3cyclictest23315-21kworker/2:022:01:322
1547299239,3cyclictest19101-21kworker/2:019:27:232
1547299238,3cyclictest23315-21kworker/2:022:47:042
1547299236,7cyclictest0-21swapper/223:03:482
1547299236,2cyclictest23315-21kworker/2:022:40:402
15472992310,3cyclictest3766-21kworker/2:220:13:402
15472992310,3cyclictest3766-21kworker/2:220:03:392
15472992310,3cyclictest23315-21kworker/2:023:57:392
15472992310,2cyclictest23315-21kworker/2:022:33:322
1547299230,16cyclictest23315-21kworker/2:023:46:172
1547299230,10cyclictest23315-21kworker/2:023:50:112
1546699237,4cyclictest24696-21pmu-power20:40:221
1546699233,15cyclictest2959-40dnf23:31:561
1546699232,11cyclictest577-21systemd-journal00:40:011
15466992310,11cyclictest21698-21kworker/u8:121:53:101
15459992310,3cyclictest6628-21kworker/0:122:30:050
1547999221,10cyclictest0-21swapper/319:20:593
1547299228,3cyclictest23315-21kworker/2:022:36:462
1547299228,12cyclictest23315-21kworker/2:021:59:252
1547299227,3cyclictest23315-21kworker/2:020:41:082
1547299226,3cyclictest3766-21kworker/2:220:37:562
1547299221,11cyclictest23315-21kworker/2:020:55:122
15472992210,3cyclictest23315-21kworker/2:021:13:392
15472992210,2cyclictest23315-21kworker/2:000:33:472
15466992213,2cyclictest4188-21diskmemload21:24:441
1546699221,18cyclictest27835-21ls00:30:131
1546699221,13cyclictest0-21swapper/122:02:141
15459992217,3cyclictest6621-21vmstat22:25:260
15459992210,4cyclictest0-21swapper/023:30:250
145312221,4sleep114543-21munin-run20:20:011
1547999219,3cyclictest13468-21sshd22:43:403
1547999212,10cyclictest16013-21/usr/sbin/munin22:50:173
15479992114,4cyclictest7261-21gltestperf21:15:163
1547999211,11cyclictest21971-21memory00:15:203
1547999211,11cyclictest15670-21/usr/sbin/munin19:10:163
1547999211,10cyclictest0-21swapper/319:37:063
1547999211,10cyclictest0-21swapper/319:33:043
1547299218,3cyclictest3766-21kworker/2:220:32:282
1547299218,3cyclictest23315-21kworker/2:022:15:562
1547299218,3cyclictest23315-21kworker/2:021:49:152
1547299211,17cyclictest19101-21kworker/2:019:50:012
1546699215,5cyclictest28000-21strings23:15:191
1546699215,4cyclictest24171-21sed23:05:241
1546699215,4cyclictest0-21swapper/122:50:271
1546699212,3cyclictest28939-21df_inode22:05:141
1546699212,10cyclictest26164-21munin-run22:00:021
1546699211,7cyclictest413-21sed23:25:251
15466992117,3cyclictest0-21swapper/123:41:551
1546699211,18cyclictest0-21swapper/121:02:291
1546699211,15cyclictest0-21swapper/121:18:161
1546699211,14cyclictest826-21lldpd22:10:581
15466992111,4cyclictest0-21swapper/121:37:551
1546699211,13cyclictest0-21swapper/121:42:401
1546699211,11cyclictest22628-21seq19:23:521
1546699211,11cyclictest19518-21diskstats20:30:151
1546699211,11cyclictest11098-21ntp_states23:50:191
1545999216,6cyclictest0-21swapper/023:35:340
1545999215,7cyclictest0-21swapper/022:47:540
1545999215,4cyclictest0-21swapper/023:50:110
1545999212,11cyclictest6607-21/usr/sbin/munin21:15:230
1545999211,8cyclictest14986-21kworker/0:121:43:500
1547999206,12cyclictest41-21ksoftirqd/322:55:173
1547999202,10cyclictest10030-21/usr/sbin/munin23:50:223
1547999201,9cyclictest0-21swapper/320:22:133
1547999201,9cyclictest0-21swapper/320:09:363
1547999201,9cyclictest0-21swapper/319:58:393
1547999201,5cyclictest0-21swapper/300:38:533
1547999201,17cyclictest0-21swapper/320:30:143
1547999201,10cyclictest26669-21sshd23:13:413
1547999201,10cyclictest16234-21sshd00:02:403
1547999201,10cyclictest0-21swapper/323:17:163
1547999201,10cyclictest0-21swapper/321:29:443
1547999200,10cyclictest0-21swapper/320:13:093
1547299206,3cyclictest23315-21kworker/2:022:22:202
1546699209,3cyclictest21220-21irqstats23:00:181
1546699204,5cyclictest4082-21/usr/sbin/munin22:20:241
15466992016,3cyclictest0-21swapper/119:37:261
1546699201,3cyclictest0-21swapper/100:25:551
15466992013,3cyclictest0-21swapper/121:28:371
1546699201,18cyclictest0-21swapper/123:35:201
1546699201,17cyclictest0-21swapper/120:05:451
1546699201,11cyclictest7662-21chrt20:01:221
1546699201,11cyclictest4726-21cstates21:10:131
1546699201,11cyclictest21822-21grep20:35:151
1546699201,11cyclictest20853-21ntp_states21:45:191
1546699201,11cyclictest19219-21/usr/sbin/munin00:10:141
1546699201,11cyclictest14770-21/usr/sbin/munin20:20:151
1546699201,10cyclictest801-21snmpd20:14:341
1546699201,10cyclictest24219-21smart_sda19:25:261
1546699201,10cyclictest15172-21munin-run21:35:011
1546699200,4cyclictest0-21swapper/100:15:271
1546699200,14cyclictest0-21swapper/122:25:401
1545999209,3cyclictest24635-21ntp_states20:40:200
1545999207,4cyclictest0-21swapper/000:33:070
1545999201,4cyclictest0-21swapper/000:14:260
1545999201,11cyclictest30790-21/usr/sbin/munin22:10:170
1545999201,10cyclictest0-21swapper/021:37:460
1545999200,6cyclictest0-21swapper/021:32:020
1545999200,15cyclictest0-21swapper/023:40:180
1547999198,3cyclictest25454-21/usr/sbin/munin00:25:223
1547999198,3cyclictest1559-21/usr/sbin/munin19:50:163
1547999196,4cyclictest20036-21taskset21:45:113
15479991914,3cyclictest22009-21cut20:35:163
1547999191,10cyclictest6077-21ntpq22:25:203
1547999191,10cyclictest22614-21/usr/sbin/munin23:05:133
1547999191,10cyclictest1885-21smart_sda22:15:233
1547999191,10cyclictest0-21swapper/321:58:113
1547999190,10cyclictest1959-21basename23:30:173
1546699192,7cyclictest0-21swapper/120:55:121
1546699192,13cyclictest10075-21gltestperf22:35:141
1546699191,9cyclictest4188-21diskmemload22:30:271
1546699191,6cyclictest27125-21timerandwakeup20:45:231
15466991916,2cyclictest0-21swapper/100:21:511
15466991912,3cyclictest0-21swapper/121:07:331
1546699191,17cyclictest0-21swapper/122:45:221
1546699191,16cyclictest0-21swapper/120:53:191
1546699191,15cyclictest607-21cat22:15:121
1546699191,11cyclictest25683-21latency23:10:171
1546699191,10cyclictest31133-21users19:40:251
1546699191,10cyclictest16741-21ntp_states19:10:231
1546699191,10cyclictest12722-21pmu-power22:40:221
15459991914,3cyclictest32388-21grep19:45:160
15459991913,4cyclictest23957-21dump-pmu-power21:55:020
15459991913,3cyclictest10016-21grep22:35:130
1545999191,10cyclictest26555-21cstates22:00:150
1545999191,10cyclictest10379-21/usr/sbin/munin20:10:170
1545999191,10cyclictest0-21swapper/000:28:260
1545999190,7cyclictest0-21swapper/022:43:470
1545999190,6cyclictest0-21swapper/023:46:590
1545999190,6cyclictest0-21swapper/022:05:260
1545999190,6cyclictest0-21swapper/021:11:060
1547999188,3cyclictest0-21swapper/321:40:273
1547999185,4cyclictest782-21in:imjournal21:02:393
1547999185,4cyclictest20921-21grep23:00:163
1547999181,9cyclictest9405-21tail21:20:173
1547999181,6cyclictest0-21swapper/323:35:153
1547999181,5cyclictest0-21swapper/323:55:233
1547999181,5cyclictest0-21swapper/322:33:403
1547999181,3cyclictest31452-21switchtime20:55:253
1547999181,3cyclictest0-21swapper/322:05:223
15479991813,3cyclictest16060-21gltestperf21:35:163
1547999181,15cyclictest24430-21grep19:25:293
1547999181,15cyclictest0-21swapper/322:45:333
1547999181,15cyclictest0-21swapper/322:01:373
1547999181,14cyclictest4188-21diskmemload23:26:133
1547999181,11cyclictest0-21swapper/321:53:273
1547999181,11cyclictest0-21swapper/320:25:293
1547999181,10cyclictest0-21swapper/321:08:053
15479991810,5cyclictest0-21swapper/323:40:263
1546699189,2cyclictest0-21swapper/119:16:481
1546699181,16cyclictest0-21swapper/123:21:011
1546699181,16cyclictest0-21swapper/119:31:561
15466991811,4cyclictest18216-21fgrep20:25:261
1545999186,4cyclictest0-21swapper/000:20:530
1545999186,3cyclictest0-21swapper/023:05:440
1545999183,8cyclictest86950irq/126-enp1s0-00:40:020
1545999183,6cyclictest14770-21/usr/sbin/munin20:20:250
1545999183,2cyclictest86950irq/126-enp1s0-21:29:070
1545999182,6cyclictest28806-21/usr/sbin/munin20:50:160
15459991813,3cyclictest10163-21awk20:10:010
15459991812,3cyclictest22946-21grep20:35:250
1545999181,15cyclictest10435-21runrttasks21:20:500
15459991811,4cyclictest32477-21cut21:00:130
1545999181,11cyclictest0-21swapper/022:50:350
1545999181,11cyclictest0-21swapper/000:08:410
1545999180,6cyclictest0-21swapper/022:23:570
223672170,2sleep00-21swapper/019:20:320
1547999177,3cyclictest86950irq/126-enp1s0-23:45:273
1547999171,6cyclictest0-21swapper/322:13:453
1547999171,6cyclictest0-21swapper/321:10:033
15479991711,4cyclictest19531-21/usr/sbin/munin00:10:163
15479991711,4cyclictest0-21swapper/320:15:143
15479991711,3cyclictest30247-21ntpq23:20:213
1547999171,10cyclictest0-21swapper/321:31:233
1547999171,10cyclictest0-21swapper/320:50:153
1547999171,10cyclictest0-21swapper/319:43:223
1547999171,10cyclictest0-21swapper/300:22:343
1546699171,15cyclictest0-21swapper/119:57:101
1545999177,3cyclictest16979-21/usr/sbin/munin20:25:250
1545999173,3cyclictest0-21swapper/020:56:490
1545999171,7cyclictest25766-21/usr/sbin/munin20:45:210
1545999171,5cyclictest0-21swapper/000:15:100
15459991712,3cyclictest25777-21users21:55:280
15459991712,3cyclictest25734-21fw_packets19:30:160
15459991712,3cyclictest20710-21grep20:30:250
15459991711,3cyclictest30466-21wc19:40:190
1545999171,13cyclictest20409-21/usr/sbin/munin23:00:220
1545999170,6cyclictest0-21swapper/000:03:250
1547999163,11cyclictest17274-21tail00:05:163
1545999161,3cyclictest3780-21/usr/sbin/munin19:55:230
15459991612,3cyclictest0-21swapper/020:02:090
15459991611,3cyclictest19201-21sed19:15:150
1545999151,2cyclictest0-21swapper/019:13:300
15459991510,3cyclictest27717-21awk19:35:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional