You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-01 - 10:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Thu Jan 01, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
168592216187,19sleep30-21swapper/319:09:583
166802203166,25sleep10-21swapper/119:07:381
165622201164,25sleep20-21swapper/219:06:062
165552201164,25sleep00-21swapper/019:05:590
2567521750,2sleep11694899cyclictest00:25:131
212482690,1sleep10-21swapper/121:45:021
166802670,2sleep20-21swapper/200:00:192
258602510,1sleep325861-21uname20:40:183
16953995047,2cyclictest0-21swapper/223:33:192
16953995046,3cyclictest0-21swapper/221:28:232
191982470,2sleep011-21rcuc/020:25:180
1694899341,27cyclictest0-21swapper/121:31:351
1694899311,1cyclictest0-21swapper/122:54:081
1694899310,25cyclictest0-21swapper/122:56:551
1694899301,4cyclictest2524-21grep22:15:141
14642300,4sleep230-21rcuc/223:25:202
92082290,4sleep230-21rcuc/223:45:022
251392290,4sleep339-21rcuc/323:05:253
192732290,4sleep230-21rcuc/219:15:012
1696099290,11cyclictest0-21swapper/323:20:123
1694899281,9cyclictest577-21systemd-journal22:40:271
1694899281,18cyclictest23196-21kworker/u8:222:21:551
1694899281,18cyclictest0-21swapper/123:57:241
1694899281,18cyclictest0-21swapper/122:26:461
16948992810,9cyclictest17266-21fschecks_count21:35:151
16948992810,2cyclictest87250irq/129-enp1s0-00:15:131
16948992810,2cyclictest87250irq/129-enp1s0-00:15:121
16948992810,15cyclictest8083-21/usr/sbin/munin21:15:201
1696099271,11cyclictest0-21swapper/322:49:133
1694899278,10cyclictest10699-21cut22:35:001
1694899271,6cyclictest27778-21cstates00:30:121
1694899271,19cyclictest0-21swapper/121:58:461
1694899271,19cyclictest0-21swapper/121:46:531
1694899271,14cyclictest0-21swapper/123:00:021
1694899271,14cyclictest0-21swapper/123:00:021
16948992710,4cyclictest11032-21grep22:35:111
1696099269,5cyclictest0-21swapper/300:30:263
1695399269,8cyclictest86950irq/126-enp1s0-22:20:162
1694899267,3cyclictest32685-21grep22:10:151
1694899261,9cyclictest0-21swapper/123:45:191
16948992614,3cyclictest0-21swapper/119:54:371
1694899261,11cyclictest0-21swapper/121:50:201
1694899261,11cyclictest0-21swapper/121:50:201
1693999262,17cyclictest4910-21/usr/sbin/munin23:35:170
265632250,1sleep30-21swapper/300:25:243
1696099251,18cyclictest0-21swapper/322:15:223
1695399258,9cyclictest0-21swapper/220:20:142
1695399251,7cyclictest0-21swapper/223:04:522
1695399251,7cyclictest0-21swapper/223:04:512
16948992521,3cyclictest0-21swapper/123:16:321
16948992521,3cyclictest0-21swapper/119:36:241
1694899251,15cyclictest0-21swapper/123:44:521
1694899251,12cyclictest6102-21kernelversion19:55:181
1694899250,15cyclictest0-21swapper/123:08:581
1696099243,14cyclictest751-21rngd19:46:173
1696099241,10cyclictest0-21swapper/321:32:083
1695399241,5cyclictest0-21swapper/223:39:372
1694899245,3cyclictest6798-21memory21:10:191
1694899242,13cyclictest87150irq/128-enp1s0-23:13:221
16948992420,3cyclictest0-21swapper/123:22:131
16948992420,3cyclictest0-21swapper/122:02:441
1694899241,9cyclictest0-21swapper/121:26:171
16948992418,3cyclictest20207-21cpuspeed_turbos19:15:111
1694899241,20cyclictest2157-21sed19:45:251
1694899241,15cyclictest0-21swapper/100:22:111
1694899241,11cyclictest5690-21/usr/sbin/munin23:35:161
1694899241,11cyclictest3054-21df_inode23:30:151
1694899241,11cyclictest23920-21ntp_states20:35:241
1694899240,5cyclictest0-21swapper/122:05:411
1696099232,9cyclictest39-21rcuc/319:25:283
1696099232,9cyclictest39-21rcuc/319:25:283
1696099231,4cyclictest0-21swapper/323:12:543
1696099231,15cyclictest0-21swapper/322:39:353
1695399235,4cyclictest28952-21cat00:30:262
1695399234,6cyclictest30-21rcuc/220:45:222
1695399231,6cyclictest9009-21kworker/u8:200:15:592
1695399231,6cyclictest9009-21kworker/u8:200:15:592
1695399231,16cyclictest0-21swapper/222:08:042
16948992320,2cyclictest0-21swapper/120:55:321
1694899231,9cyclictest0-21swapper/100:14:411
1694899231,3cyclictest15817-21grep22:45:171
16948992311,3cyclictest18136-21seq00:05:271
1694899231,11cyclictest1852-21cat23:25:251
16948992310,3cyclictest2739-21smart_sda21:00:251
1694899230,3cyclictest0-21swapper/123:51:431
1694899230,3cyclictest0-21swapper/100:36:011
1693999238,3cyclictest30421-1kworker/u9:200:04:010
1693999231,7cyclictest8109-21kworker/u8:022:28:280
1696099226,8cyclictest0-21swapper/320:14:333
1696099223,16cyclictest401ktimersoftd/323:55:213
1696099223,12cyclictest0-21swapper/300:15:483
1696099223,12cyclictest0-21swapper/300:15:483
16960992210,3cyclictest8518-21cat22:30:023
1696099220,8cyclictest0-21swapper/322:59:343
1695399224,6cyclictest30-21rcuc/222:10:272
1695399224,5cyclictest10475-21proc_pri23:45:232
1695399221,8cyclictest0-21swapper/222:43:222
1695399221,15cyclictest0-21swapper/220:59:312
1695399221,15cyclictest0-21swapper/219:16:232
1695399220,16cyclictest0-21swapper/219:48:342
1694899224,3cyclictest25910-21grep19:25:271
1694899224,3cyclictest25910-21grep19:25:271
16948992219,2cyclictest0-21swapper/121:06:151
16948992219,2cyclictest0-21swapper/120:42:411
16948992219,2cyclictest0-21swapper/120:12:241
16948992218,3cyclictest0-21swapper/121:24:101
16948992218,3cyclictest0-21swapper/120:49:571
1694899221,3cyclictest16965-21ntpq00:00:231
1694899221,18cyclictest17955-21interrupts19:10:171
1694899220,20cyclictest0-21swapper/119:30:221
1693999221,8cyclictest27626-21latency_hist22:00:010
1693999221,5cyclictest0-21swapper/022:15:270
1693999221,3cyclictest5665-21awk21:10:020
1696099213,10cyclictest39-21rcuc/323:40:183
1696099213,10cyclictest0-21swapper/323:30:273
1696099212,12cyclictest0-21swapper/321:43:243
1696099211,4cyclictest0-21swapper/321:18:293
1696099211,10cyclictest4648-21diskstats22:20:143
1695399218,4cyclictest26116-21ntp_states20:40:212
1695399216,3cyclictest86950irq/126-enp1s0-19:29:312
1695399216,3cyclictest86950irq/126-enp1s0-19:29:312
1695399215,8cyclictest32-21ksoftirqd/222:30:182
1695399215,4cyclictest86950irq/126-enp1s0-21:40:242
1695399214,5cyclictest30-21rcuc/223:20:242
1695399212,5cyclictest27370-21ntp_states23:10:372
1695399211,9cyclictest0-21swapper/200:35:172
1695399211,7cyclictest0-21swapper/222:58:512
1695399211,6cyclictest7323-21cat20:00:022
1695399211,4cyclictest0-21swapper/223:57:392
16953992114,4cyclictest20212-21grep00:10:242
1695399211,2cyclictest87150irq/128-enp1s0-21:36:522
16948992117,3cyclictest0-21swapper/120:25:431
16948992117,3cyclictest0-21swapper/120:06:071
1694899211,10cyclictest0-21swapper/120:20:451
1693999211,8cyclictest0-21swapper/023:11:460
1693999211,8cyclictest0-21swapper/022:32:320
1693999211,4cyclictest31408-21sh20:55:010
1696099205,5cyclictest21537-21kworker/u8:100:20:123
1696099204,5cyclictest26319-21kworker/u8:121:12:373
1696099203,7cyclictest0-21swapper/321:50:203
1696099203,7cyclictest0-21swapper/321:50:203
1696099201,9cyclictest557-21awk19:45:013
1696099201,7cyclictest0-21swapper/320:51:303
1696099201,6cyclictest29410-21perl19:35:183
1696099201,4cyclictest0-21swapper/323:26:123
1696099201,4cyclictest0-21swapper/322:42:553
1696099201,2cyclictest86950irq/126-enp1s0-22:05:133
16960992011,7cyclictest26319-21kworker/u8:119:56:343
1696099201,10cyclictest0-21swapper/323:00:263
1696099201,10cyclictest0-21swapper/323:00:253
1696099201,10cyclictest0-21swapper/300:04:563
1696099200,9cyclictest0-21swapper/321:48:283
1696099200,9cyclictest0-21swapper/300:08:203
1696099200,12cyclictest0-21swapper/323:15:233
1696099200,10cyclictest0-21swapper/323:50:273
1696099200,10cyclictest0-21swapper/321:59:583
1695399206,3cyclictest86950irq/126-enp1s0-19:35:182
1695399205,3cyclictest12050irq/123-ahci[0023:20:012
1695399204,4cyclictest782-21in:imjournal00:23:412
1695399203,5cyclictest13672-1kworker/2:0H21:48:372
1695399201,7cyclictest0-21swapper/221:15:552
1695399201,6cyclictest0-21swapper/221:11:182
1695399201,6cyclictest0-21swapper/220:25:472
1695399201,4cyclictest0-21swapper/223:06:462
1695399200,4cyclictest0-21swapper/221:59:532
1694899201,9cyclictest0-21swapper/120:30:201
1694899201,6cyclictest22240-21/usr/sbin/munin19:20:161
1694899201,18cyclictest0-21swapper/119:42:281
1694899201,17cyclictest0-21swapper/120:15:201
1693999209,2cyclictest30421-1kworker/u9:222:00:010
1693999205,8cyclictest21272-21/usr/sbin/munin21:45:220
1693999204,4cyclictest25194-21cat00:25:020
1693999204,3cyclictest9047-21kworker/0:020:08:050
1693999203,9cyclictest3311-21kworker/0:121:10:260
1693999201,7cyclictest2295-21chrt19:45:240
1693999201,6cyclictest26059-21munin-run23:10:010
1693999201,5cyclictest13555-21tune2fs22:40:160
1693999201,4cyclictest0-21swapper/023:25:140
1693999201,11cyclictest22703-21sshd00:17:310
1693999201,11cyclictest22703-21sshd00:17:300
1693999201,11cyclictest0-21swapper/021:17:140
1696099199,3cyclictest801-21snmpd20:30:123
1696099198,3cyclictest801-21snmpd19:50:053
1696099196,7cyclictest23196-21kworker/u8:222:30:153
1696099196,3cyclictest0-21swapper/300:10:553
1696099195,4cyclictest0-21swapper/321:20:513
1696099195,3cyclictest0-21swapper/321:25:263
1696099192,2cyclictest87150irq/128-enp1s0-00:36:343
1696099191,7cyclictest0-21swapper/321:35:433
16960991913,3cyclictest22880-21tail19:20:163
1696099191,11cyclictest0-21swapper/322:14:023
1696099191,11cyclictest0-21swapper/320:45:133
1696099191,10cyclictest26632-21/usr/sbin/munin19:30:153
1696099191,10cyclictest17488-21/usr/sbin/munin22:50:263
1696099190,9cyclictest0-21swapper/323:46:343
1695399196,3cyclictest87250irq/129-enp1s0-20:15:212
1695399195,3cyclictest86950irq/126-enp1s0-22:50:122
1695399194,3cyclictest5240-21sed21:05:262
1695399194,3cyclictest24452-21needreboot21:50:172
1695399194,3cyclictest24452-21needreboot21:50:172
1695399193,3cyclictest12327-21cat21:25:012
1695399192,5cyclictest32-21ksoftirqd/220:00:272
1695399192,4cyclictest14853-21cstates21:30:142
1695399191,7cyclictest32-21ksoftirqd/221:00:132
1695399191,7cyclictest0-21swapper/222:45:182
1695399191,6cyclictest29840-21cut20:50:152
1695399191,6cyclictest0-21swapper/222:26:132
1695399191,5cyclictest4071-21latency_hist22:20:022
1695399191,5cyclictest12537-21hddtemp_smartct20:10:192
16953991914,3cyclictest27271-21gltestperf19:30:162
1694899191,15cyclictest7517-21/usr/sbin/munin20:00:131
1694899190,9cyclictest0-21swapper/120:51:321
1693999197,3cyclictest32173-21pmu-power19:40:250
1693999195,7cyclictest17908-21if_enp2s019:10:170
1693999195,3cyclictest30421-1kworker/u9:223:42:400
1693999194,7cyclictest21967-21grep23:00:130
1693999194,7cyclictest21967-21grep23:00:130
1693999193,8cyclictest15586-21memory21:30:190
1693999193,3cyclictest87150irq/128-enp1s0-00:37:460
1693999192,3cyclictest18386-21sshd00:08:300
1693999191,5cyclictest26094-21if_enp2s000:25:180
1693999191,10cyclictest0-21swapper/022:23:450
1693999191,10cyclictest0-21swapper/000:10:410
45182181,2sleep00-21swapper/023:33:010
263072181,2sleep00-21swapper/019:28:560
263072181,2sleep00-21swapper/019:28:560
1696099188,3cyclictest3686-21/usr/sbin/munin21:05:193
1696099188,2cyclictest86950irq/126-enp1s0-20:37:453
1696099185,3cyclictest0-21swapper/320:19:453
1696099183,2cyclictest86950irq/126-enp1s0-23:35:233
1696099181,15cyclictest0-21swapper/321:00:133
16960991810,5cyclictest0-21swapper/320:20:203
1696099180,10cyclictest0-21swapper/320:55:123
1695399183,5cyclictest0-21swapper/222:36:292
1695399183,3cyclictest801-21snmpd19:50:052
1695399181,7cyclictest0-21swapper/200:25:522
1695399181,6cyclictest0-21swapper/222:01:462
1695399181,5cyclictest22240-21/usr/sbin/munin19:20:142
16953991813,3cyclictest9663-21/usr/sbin/munin20:05:122
1693999186,3cyclictest31436-21if_enp2s023:20:170
1693999185,3cyclictest14058-21/usr/sbin/munin20:15:150
1693999184,7cyclictest10577-21swap23:45:230
1693999184,4cyclictest17546-21gltestperf21:35:150
1693999183,8cyclictest7458-21/usr/sbin/munin20:00:150
1693999182,7cyclictest0-21swapper/022:55:220
1693999181,9cyclictest801-21snmpd22:50:380
1693999181,9cyclictest29159-21taskset00:30:470
1693999181,9cyclictest13723-21cpu23:55:100
1693999181,8cyclictest0-21swapper/021:25:150
1693999181,8cyclictest0-21swapper/021:24:370
1693999181,6cyclictest31656-21cpu20:55:130
1693999181,6cyclictest23591-21cpuspeed_turbos21:50:090
1693999181,6cyclictest23591-21cpuspeed_turbos21:50:090
1693999181,4cyclictest29474-21cut23:15:220
16939991813,3cyclictest32267-21/usr/sbin/munin22:10:240
1693999181,14cyclictest16252-21/usr/sbin/munin20:20:160
1693999181,11cyclictest0-21swapper/022:45:270
1696099176,8cyclictest39-21rcuc/319:10:223
1696099176,3cyclictest86950irq/126-enp1s0-20:05:213
1696099174,4cyclictest28654-21cat22:00:183
1696099171,5cyclictest788-21NetworkManager20:27:583
1696099170,7cyclictest86950irq/126-enp1s0-19:15:113
1695399174,2cyclictest86950irq/126-enp1s0-19:40:142
1695399173,3cyclictest87250irq/129-enp1s0-20:30:122
1695399171,5cyclictest23088-21expr20:35:142
1695399171,4cyclictest0-21swapper/200:06:312
1695399170,6cyclictest30-21rcuc/223:50:192
1693999179,3cyclictest2630-21proc_pri21:00:240
1693999175,4cyclictest0-21swapper/020:45:120
1693999174,3cyclictest25072-21cpu20:40:120
1693999172,5cyclictest3060-21/usr/sbin/munin19:50:200
1693999172,2cyclictest12492-21grep23:50:200
1693999171,6cyclictest19138-21cpuspeed_turbos21:40:120
1693999171,5cyclictest0-21swapper/022:39:320
1693999171,3cyclictest0-21swapper/022:05:200
16939991712,3cyclictest29624-21if_err_enp2s019:35:190
16939991711,4cyclictest22657-21grep19:20:160
16939991711,3cyclictest19542-21ntpq19:15:020
1693999171,13cyclictest12852-21head20:10:220
1696099161,14cyclictest0-21swapper/320:01:493
1693999161,6cyclictest23575-21basename20:35:190
1693999161,3cyclictest0-21swapper/020:30:260
16939991611,3cyclictest5559-21cat19:55:130
1693999151,3cyclictest26893-21expr19:30:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional