You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-22 - 23:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sun Mar 22, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
55842237162,24sleep10-21swapper/107:08:271
56952213163,38sleep20-21swapper/207:09:532
53592201163,25sleep30-21swapper/307:05:323
37422201164,24sleep00-21swapper/007:05:050
711021720,5sleep1580099cyclictest12:05:121
711021720,5sleep1580099cyclictest12:05:121
788621430,2sleep0579399cyclictest08:25:150
580999107104,2cyclictest0-21swapper/311:46:393
58099910399,3cyclictest0-21swapper/309:28:163
58099910297,3cyclictest0-21swapper/308:45:453
5809999794,2cyclictest0-21swapper/309:43:023
80712540,4sleep1580099cyclictest07:15:021
80712540,4sleep1580099cyclictest07:15:011
157452520,2sleep081ktimersoftd/012:25:030
43452500,2sleep30-21swapper/310:45:163
5809994642,3cyclictest0-21swapper/312:21:493
5809994138,2cyclictest0-21swapper/309:02:213
5809993632,3cyclictest0-21swapper/307:47:113
580499363,8cyclictest2335-40dnf08:10:502
580999343,23cyclictest0-21swapper/310:09:053
580999330,24cyclictest19293-21kworker/u8:211:31:393
579399336,12cyclictest24659-21idleruntime-cro09:05:010
580999323,21cyclictest86550irq/126-enp1s0-09:22:473
580999320,25cyclictest8006-21kworker/u8:011:21:263
580999320,24cyclictest31178-21kworker/u8:109:30:443
580999312,22cyclictest86550irq/126-enp1s0-12:02:033
580999311,24cyclictest0-21swapper/312:31:213
580499311,3cyclictest0-21swapper/211:36:102
580999303,13cyclictest16862-40dnf11:11:593
579399304,12cyclictest9908-21awk09:45:010
579399304,12cyclictest15272-21perf07:30:020
579399301,14cyclictest14919-21munin-run11:10:010
579399301,13cyclictest13679-21munin-run08:40:010
579399300,14cyclictest13353-21awk12:20:020
580999290,5cyclictest0-21swapper/310:39:433
580499291,5cyclictest8876-21sh10:55:122
580499291,14cyclictest0-21swapper/208:44:322
580099290,22cyclictest0-21swapper/110:55:161
5804992818,2cyclictest0-21swapper/212:10:222
5804992810,3cyclictest0-21swapper/211:15:132
580099281,20cyclictest0-21swapper/109:15:101
579399289,16cyclictest21838-21idleruntime-cro07:45:020
579399281,15cyclictest8949-21munin-run12:10:010
579399281,15cyclictest8949-21munin-run12:10:010
5809992724,2cyclictest0-21swapper/307:39:013
580999271,23cyclictest1698-21head08:10:193
580499271,5cyclictest1262-21memory11:50:192
580099271,17cyclictest0-21swapper/112:20:041
222292270,4sleep121-21rcuc/110:10:221
580999262,21cyclictest39-21rcuc/308:20:163
580999261,22cyclictest401ktimersoftd/309:15:213
580999261,22cyclictest26871-21diskmemload11:00:133
580999261,17cyclictest0-21swapper/309:37:523
580499268,9cyclictest0-21swapper/210:19:482
580499261,13cyclictest0-21swapper/209:23:282
580499261,12cyclictest18634-21hddtemp_smartct12:30:162
580499261,12cyclictest0-21swapper/211:47:362
580499261,12cyclictest0-21swapper/211:11:212
580499261,11cyclictest0-21swapper/211:05:052
580099263,16cyclictest86750irq/128-enp1s0-10:40:221
579399261,12cyclictest17117-21munin-run11:15:010
580999251,22cyclictest0-21swapper/310:04:413
580999251,22cyclictest0-21swapper/307:50:213
580999251,21cyclictest26974-21unixbench-2d10:20:253
580999251,21cyclictest21149-21cut12:35:203
580999251,16cyclictest0-21swapper/311:54:483
580499259,3cyclictest0-21swapper/208:35:122
580499251,12cyclictest0-21swapper/211:59:402
580099251,3cyclictest11480-21munin-run08:35:021
580099251,3cyclictest11480-21munin-run08:35:011
580099251,16cyclictest0-21swapper/111:54:271
580099251,14cyclictest0-21swapper/111:15:151
580099250,20cyclictest0-21swapper/109:25:291
27822250,5sleep341-21ksoftirqd/310:40:183
580999241,21cyclictest30211-21kworker/u8:212:25:213
580999241,21cyclictest0-21swapper/310:50:193
580999241,20cyclictest0-21swapper/312:05:393
580999241,20cyclictest0-21swapper/312:05:383
580999241,20cyclictest0-21swapper/310:34:283
580999241,20cyclictest0-21swapper/310:18:183
580999241,20cyclictest0-21swapper/310:13:443
580999241,20cyclictest0-21swapper/309:46:473
580999241,15cyclictest0-21swapper/310:56:133
580999241,12cyclictest0-21swapper/311:25:363
580999241,11cyclictest7145-21awk07:10:233
580999241,11cyclictest7145-21awk07:10:233
580999241,11cyclictest0-21swapper/312:17:363
580499249,4cyclictest24008-21switchtime09:00:242
580499248,3cyclictest0-21swapper/209:07:312
580499247,4cyclictest18009-21df07:35:112
580499247,3cyclictest0-21swapper/210:47:162
580499247,3cyclictest0-21swapper/207:20:072
580499241,11cyclictest0-21swapper/212:03:002
580499241,10cyclictest0-21swapper/212:23:162
580499241,10cyclictest0-21swapper/207:18:552
580499241,10cyclictest0-21swapper/207:18:552
5804992410,3cyclictest0-21swapper/209:40:122
5804992410,3cyclictest0-21swapper/208:06:592
5800992419,3cyclictest5181-21uniq12:00:141
580099241,3cyclictest23468-21if_err_enp2s009:00:171
580099241,16cyclictest0-21swapper/111:07:211
580099240,18cyclictest28733-21kworker/u8:310:26:251
579399241,12cyclictest30315-21awk11:45:000
579399241,12cyclictest19327-21basename11:20:010
580999232,10cyclictest0-21swapper/309:50:243
580999231,8cyclictest15456-21sed08:40:253
580999231,21cyclictest0-21swapper/308:50:283
580999231,20cyclictest28988-21kworker/u8:208:28:323
580999231,20cyclictest0-21swapper/311:42:063
580999231,20cyclictest0-21swapper/309:11:013
580999231,19cyclictest401ktimersoftd/311:15:113
580999231,19cyclictest0-21swapper/309:55:593
580999231,10cyclictest26868-21munin-run09:10:013
580999231,10cyclictest13283-21/usr/sbin/munin07:25:213
580499239,6cyclictest21431-21missed_timers08:55:202
580499238,3cyclictest22609-21sed11:25:212
580499238,3cyclictest0-21swapper/209:14:042
580499238,3cyclictest0-21swapper/207:43:362
580499237,3cyclictest0-21swapper/208:53:052
580499231,12cyclictest0-21swapper/209:34:082
580499231,11cyclictest26871-21diskmemload10:51:082
580499231,10cyclictest0-21swapper/212:07:162
580499231,10cyclictest0-21swapper/212:07:162
580499231,10cyclictest0-21swapper/211:43:402
5804992310,3cyclictest13422-21/usr/sbin/munin09:50:222
580099238,3cyclictest27479-21fschecks_count09:10:141
580099231,7cyclictest24280-21diskstats11:30:131
5800992317,4cyclictest22711-21sed07:45:171
580099231,6cyclictest0-21swapper/110:33:361
5800992314,2cyclictest27039-21cat09:10:021
580099231,3cyclictest86750irq/128-enp1s0-11:00:131
580099231,16cyclictest0-21swapper/109:56:041
580099231,13cyclictest0-21swapper/110:08:381
580099231,12cyclictest12738-21cut09:50:151
580099231,11cyclictest0-21swapper/110:23:381
580099231,10cyclictest26871-21diskmemload10:45:161
580099230,6cyclictest0-21swapper/110:02:441
580099230,17cyclictest0-21swapper/109:22:481
580099230,16cyclictest0-21swapper/111:41:301
579399231,5cyclictest0-21swapper/010:43:200
580999221,19cyclictest0-21swapper/311:05:223
580999221,19cyclictest0-21swapper/308:56:413
580999221,10cyclictest3379-21kernelversion11:55:183
580499229,3cyclictest0-21swapper/212:28:322
580499228,3cyclictest0-21swapper/211:03:532
580499227,3cyclictest0-21swapper/210:31:082
580499227,3cyclictest0-21swapper/210:27:442
580499227,3cyclictest0-21swapper/210:09:282
580499227,3cyclictest0-21swapper/209:26:242
580499227,3cyclictest0-21swapper/208:32:032
580499227,3cyclictest0-21swapper/208:32:032
580499226,3cyclictest3617-21taskset10:41:202
580499226,3cyclictest15581-21ntp_states09:55:202
580499221,13cyclictest0-21swapper/208:19:342
580499221,11cyclictest0-21swapper/211:34:202
580099226,9cyclictest0-21swapper/109:31:381
580099225,5cyclictest28734-1kworker/u9:211:27:311
580099224,9cyclictest0-21swapper/112:36:401
580099222,12cyclictest0-21swapper/112:25:261
5800992218,3cyclictest0-21swapper/110:15:221
580099221,3cyclictest18309-21diskstats12:30:141
580099221,16cyclictest0-21swapper/110:54:091
580099221,15cyclictest0-21swapper/109:45:091
580099221,12cyclictest0-21swapper/110:39:341
580099221,11cyclictest21124-21users11:20:251
580099221,11cyclictest0-21swapper/109:35:031
580099221,10cyclictest812-21snmpd11:14:001
580099220,7cyclictest0-21swapper/111:35:311
580099220,16cyclictest0-21swapper/112:14:121
579399226,15cyclictest0-21swapper/011:00:010
579399222,11cyclictest21737-21/usr/sbin/munin11:25:140
313062221,3sleep031383-21latency11:45:180
288362221,3sleep328951-21sendmail_mailst10:25:223
580999214,14cyclictest39-21rcuc/308:35:243
580999211,9cyclictest10232-21ntpq12:10:223
580499218,3cyclictest0-21swapper/209:17:162
580499217,3cyclictest0-21swapper/207:52:002
580499216,3cyclictest0-21swapper/209:39:322
580499211,8cyclictest22185-21ntp_states10:10:222
5804992116,3cyclictest27062-21grep10:20:262
580099212,10cyclictest1240-21perl08:10:151
580099211,7cyclictest11702-21/usr/sbin/munin08:35:131
5800992117,3cyclictest0-21swapper/108:40:141
5800992117,3cyclictest0-21swapper/107:18:491
5800992117,3cyclictest0-21swapper/107:18:491
5800992116,3cyclictest15589-21grep07:30:111
5800992115,3cyclictest18075-21perf08:50:011
580099211,18cyclictest0-21swapper/108:27:061
580099211,18cyclictest0-21swapper/108:01:001
580099211,15cyclictest0-21swapper/111:48:351
580099211,15cyclictest0-21swapper/109:42:271
580099211,15cyclictest0-21swapper/107:37:091
580099211,11cyclictest2910-21diskstats11:55:131
580099211,11cyclictest21555-21ntp_states08:55:211
5800992110,3cyclictest4881-21munin-run08:20:021
579399219,6cyclictest0-21swapper/011:01:060
5793992117,3cyclictest7-21ksoftirqd/012:12:480
580999201,5cyclictest10863-21munin-run07:20:013
580999201,5cyclictest10863-21munin-run07:20:013
580999201,11cyclictest0-21swapper/308:05:023
580999201,10cyclictest0-21swapper/307:57:573
580999201,10cyclictest0-21swapper/307:31:293
580499207,3cyclictest0-21swapper/212:15:362
580499207,3cyclictest0-21swapper/210:03:162
580499207,3cyclictest0-21swapper/209:45:402
580499207,2cyclictest0-21swapper/211:20:252
580499206,3cyclictest0-21swapper/208:29:152
580499205,3cyclictest0-21swapper/207:50:002
580499204,3cyclictest0-21swapper/210:38:042
5800992016,3cyclictest0-21swapper/108:09:531
5800992014,3cyclictest5781-21hddtemp_smartct08:20:171
580099201,18cyclictest0-21swapper/107:41:501
580099201,17cyclictest0-21swapper/108:54:141
5800992010,3cyclictest26237-21latency_hist07:55:011
580099200,15cyclictest0-21swapper/112:15:161
579399201,13cyclictest0-21swapper/009:20:110
579399201,11cyclictest812-21snmpd09:12:400
579399201,11cyclictest2907-21/usr/sbin/munin08:15:250
579399201,11cyclictest23933-21/usr/sbin/munin11:30:170
579399201,11cyclictest0-21swapper/010:06:300
579399201,10cyclictest29912-21/usr/sbin/munin10:30:130
579399201,10cyclictest17780-21ntp_states10:00:210
580999195,3cyclictest86550irq/126-enp1s0-08:00:123
580999192,14cyclictest39-21rcuc/307:40:163
580999191,5cyclictest0-21swapper/311:38:263
580999191,3cyclictest0-21swapper/308:15:193
580499194,3cyclictest0-21swapper/207:10:042
580499194,3cyclictest0-21swapper/207:10:032
580099193,3cyclictest26852-21fschecks_count07:55:141
579399198,3cyclictest30868-21/usr/sbin/munin08:05:170
579399196,5cyclictest81ktimersoftd/010:50:190
579399193,7cyclictest609-21/usr/sbin/munin08:10:240
579399192,5cyclictest11-21rcuc/009:30:010
579399192,14cyclictest23585-21grep07:45:280
5793991914,3cyclictest7377-21cat07:10:260
5793991914,3cyclictest7377-21cat07:10:260
579399191,15cyclictest5589-21sed12:00:180
579399191,12cyclictest0-21swapper/010:39:120
579399191,10cyclictest4202-21sshd11:55:570
579399191,10cyclictest4147-21/usr/sbin/munin10:45:260
579399191,10cyclictest24893-21/usr/sbin/munin09:05:270
579399191,10cyclictest19538-21/usr/sbin/munin11:20:200
579399190,8cyclictest0-21swapper/009:37:570
290802190,4sleep011-21rcuc/008:00:150
196352191,2sleep00-21swapper/007:40:020
580999181,7cyclictest11433-21diskstats07:20:133
580999181,5cyclictest11639-21cat08:35:013
580999181,5cyclictest11639-21cat08:35:013
580499181,14cyclictest20253-21grep12:35:132
580499181,13cyclictest0-21swapper/208:23:392
580099181,4cyclictest0-21swapper/107:20:111
580099181,16cyclictest0-21swapper/107:25:041
579399181,8cyclictest0-21swapper/007:15:130
579399181,8cyclictest0-21swapper/007:15:130
579399181,5cyclictest30280-21strings09:15:180
579399181,3cyclictest17743-21perf12:30:020
579399181,10cyclictest0-21swapper/012:32:270
19552180,3sleep00-21swapper/011:50:270
580499171,6cyclictest14345-21ntp_states07:25:232
5804991712,3cyclictest29893-21sendmail_mailqu08:00:232
580499171,13cyclictest0-21swapper/208:47:532
5804991710,5cyclictest0-21swapper/207:30:122
580499170,3cyclictest27107-21gltestperf07:55:152
579399178,7cyclictest15696-21sshd08:42:570
579399175,5cyclictest0-21swapper/009:59:390
579399174,4cyclictest7-21ksoftirqd/009:50:150
579399172,12cyclictest11015-21grep08:30:240
579399172,12cyclictest11015-21grep08:30:240
579399171,6cyclictest580-21systemd-journal09:35:010
579399171,6cyclictest0-21swapper/010:11:220
579399171,3cyclictest26129-21/usr/sbin/munin11:35:110
5793991713,3cyclictest7-21ksoftirqd/010:25:530
5793991713,3cyclictest0-21swapper/008:21:430
5793991712,3cyclictest12623-21unixbench_singl07:20:240
5793991711,4cyclictest24449-21head10:15:230
579399170,15cyclictest26253-21awk07:55:010
272512171,2sleep00-21swapper/010:22:140
119602170,3sleep00-21swapper/009:48:310
579399161,6cyclictest26465-21/usr/sbin/munin07:55:230
579399161,6cyclictest0-21swapper/008:50:160
5793991612,3cyclictest0-21swapper/012:38:100
5793991611,3cyclictest21540-21ntp_offset08:55:200
5793991610,3cyclictest17366-21seq07:34:070
5793991610,3cyclictest16832-21awk08:45:180
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional