You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-17 - 11:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Wed Jun 17, 2026 00:44:45)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
91609121270,3sleep00-21swapper/000:05:150
91234421150,4sleep0912323-21perf23:55:000
88855721100,2sleep13056-21PK-Backend22:45:201
89165421000,4sleep126-21ksoftirqd/122:55:141
8593872980,3sleep0161rcuc/021:23:030
81156928439,14sleep20-21swapper/219:08:482
139099740,64rtkit-daemon0-21swapper/119:09:311
81160727140,25sleep00-21swapper/019:09:190
81144926937,26sleep30-21swapper/319:07:163
8709112650,4sleep10-21swapper/121:55:151
8198562440,2sleep13056-21PK-Backend19:30:161
811811993111,6cyclictest904907-21anacron23:34:590
81181899300,2cyclictest826087-21apt-get19:50:032
9040542290,4sleep3904059-21needreboot23:30:173
81181899288,18cyclictest919633-21ls00:15:122
81181899288,18cyclictest909007-21dpkg23:45:002
81181899282,22cyclictest978-21snmpd20:25:122
81181899282,21cyclictest0-21swapper/222:55:132
81181199281,8cyclictest851156-21apt21:00:100
811811992813,3cyclictest917969-21tr00:10:120
8864862270,3chrt0-21swapper/322:40:163
81181899278,17cyclictest880115-21sed22:20:212
81181899277,18cyclictest888134-21expr22:45:132
81181899276,18cyclictest878455-21timerwakeupswit22:15:242
81181899272,20cyclictest0-21swapper/223:05:142
81181899271,21cyclictest861782-21apt-get21:30:032
81181899270,7cyclictest0-21swapper/223:35:172
81181899268,16cyclictest860735-21switchtime21:25:232
81181899266,18cyclictest925165-21sort00:30:172
81181899265,18cyclictest867383-21/usr/sbin/munin21:45:122
81181899265,17cyclictest904272-21python323:30:202
81181899262,20cyclictest0-21swapper/200:25:132
81181899262,20cyclictest0-21swapper/200:25:132
81181899262,19cyclictest0-21swapper/219:35:112
81181899261,23cyclictest841973-21munin-run20:35:002
81181899261,21cyclictest909095-21apt-get23:45:032
81181899260,4cyclictest894-21irqbalance00:35:192
139099260,4rtkit-daemon1389-21rtkit-daemon22:09:171
139099260,4rtkit-daemon1389-21rtkit-daemon20:55:231
81181899259,14cyclictest872300-21cpuspeed22:00:132
81181899259,14cyclictest0-21swapper/221:55:152
81181899257,16cyclictest824221-21seq19:43:302
81181899256,17cyclictest900037-21ls23:20:112
81181899255,18cyclictest819650-21cat19:30:132
81181899251,22cyclictest847858-21grep20:50:122
81181899251,19cyclictest814226-21apt19:15:112
811818992510,12cyclictest917980-21cpuspeed_turbos00:10:112
811811992519,5cyclictest0-21swapper/020:29:350
81181899249,13cyclictest916070-21acpi00:05:142
81181899248,14cyclictest0-21swapper/221:35:182
81181899246,16cyclictest0-21swapper/221:20:002
81181899244,5cyclictest853099-21cat21:05:132
81181899242,17cyclictest824626-21apt19:45:122
81181899241,7cyclictest847455-21unin-run20:50:002
81181899241,19cyclictest0-21swapper/221:40:112
81181899241,18cyclictest898209-21apt23:15:122
81181899240,22cyclictest891082-21cat22:55:002
81181199240,1cyclictest0-21swapper/022:13:170
81181199240,19cyclictest0-21swapper/019:10:400
811822992321,1cyclictest3056-21PK-Backend22:02:273
811822992318,3cyclictest848063-21df_inode20:50:143
811822992316,4cyclictest860686-21python321:25:233
81181899235,16cyclictest869153-21smartctl21:50:152
81181899234,5cyclictest921437-21grep00:20:132
81181899234,17cyclictest0-21swapper/200:02:122
81181899231,5cyclictest896474-21/usr/sbin/munin23:10:112
81181899230,3cyclictest838526-21munin-run20:24:592
811811992321,1cyclictest0-21swapper/019:21:420
139099230,4rtkit-daemon1389-21rtkit-daemon23:50:091
811822992220,1cyclictest3056-21PK-Backend20:06:143
811822992220,1cyclictest0-21swapper/300:06:433
81182299221,8cyclictest871061-21memory21:55:173
81181899228,12cyclictest881090-21cstates22:25:142
81181899226,14cyclictest876679-21cat22:10:202
81181899226,14cyclictest858300-21cstates21:20:132
81181899226,14cyclictest0-21swapper/220:00:012
81181899225,16cyclictest852511-21seq21:03:462
81181899222,18cyclictest874810-21python322:05:232
81181899221,4cyclictest819145-21cron19:30:002
81181899221,4cyclictest436-20systemd-journal23:30:002
81181899221,4cyclictest436-20systemd-journal23:29:592
81181899221,4cyclictest436-20systemd-journal20:17:002
81181899221,4cyclictest1-21systemd22:45:002
81181899221,4cyclictest0-21swapper/221:10:132
81181899221,3cyclictest0-21swapper/223:00:232
81181899220,11cyclictest0-21swapper/220:45:002
811815992220,1cyclictest3056-21PK-Backend23:39:501
81181199222,18cyclictest870223-21munin-run21:55:000
811811992220,1cyclictest0-21swapper/023:24:290
811811992220,1cyclictest0-21swapper/019:37:080
811811992220,1cyclictest0-21swapper/019:17:090
139099220,19rtkit-daemon1997-21gvfs-afc-volume22:30:111
811822992119,1cyclictest0-21swapper/322:20:113
811822992119,1cyclictest0-21swapper/322:15:383
811822992119,1cyclictest0-21swapper/321:38:133
811822992119,1cyclictest0-21swapper/321:16:443
811822992119,1cyclictest0-21swapper/319:48:023
811822992119,1cyclictest0-21swapper/319:33:223
811822992119,1cyclictest0-21swapper/300:19:223
811822992119,1cyclictest0-21swapper/300:10:093
811822992119,1cyclictest0-21swapper/300:02:313
811822992118,2cyclictest3056-21PK-Backend20:37:373
811822992113,6cyclictest436-20systemd-journal00:39:233
81182299210,20cyclictest0-21swapper/323:03:103
81182299210,20cyclictest0-21swapper/322:45:123
81182299210,19cyclictest3056-21PK-Backend22:35:113
81182299210,19cyclictest3056-21PK-Backend19:57:013
81182299210,19cyclictest3056-21PK-Backend19:26:393
81181899211,5cyclictest883-21dbus-daemon20:03:312
81181899211,4cyclictest849423-21/usr/sbin/munin20:55:122
81181899210,4cyclictest885608-21sshd22:37:352
81181599219,9cyclictest883-21dbus-daemon19:20:011
811815992119,1cyclictest0-21swapper/100:20:111
811815992110,9cyclictest3056-21PK-Backend21:45:001
811811992119,1cyclictest0-21swapper/023:56:590
811811992119,1cyclictest0-21swapper/023:41:120
811811992119,1cyclictest0-21swapper/023:37:120
811811992119,1cyclictest0-21swapper/022:58:400
811811992119,1cyclictest0-21swapper/022:43:050
811811992119,1cyclictest0-21swapper/022:31:290
811811992119,1cyclictest0-21swapper/022:29:540
811811992119,1cyclictest0-21swapper/021:38:010
811811992119,1cyclictest0-21swapper/021:31:540
811811992119,1cyclictest0-21swapper/021:08:280
811811992119,1cyclictest0-21swapper/020:42:590
811811992119,1cyclictest0-21swapper/020:33:000
811811992119,1cyclictest0-21swapper/019:27:230
811811992119,1cyclictest0-21swapper/000:30:450
811811992113,6cyclictest436-20systemd-journal20:45:320
81181199210,20cyclictest0-21swapper/021:48:570
81181199210,20cyclictest0-21swapper/021:19:570
81181199210,20cyclictest0-21swapper/020:21:580
81181199210,20cyclictest0-21swapper/020:16:100
81181199210,20cyclictest0-21swapper/019:45:550
811822992018,1cyclictest3056-21PK-Backend20:46:023
811822992018,1cyclictest0-21swapper/323:40:083
811822992018,1cyclictest0-21swapper/323:36:083
811822992018,1cyclictest0-21swapper/323:20:273
811822992018,1cyclictest0-21swapper/323:16:283
811822992018,1cyclictest0-21swapper/323:14:043
811822992018,1cyclictest0-21swapper/321:32:533
811822992018,1cyclictest0-21swapper/319:50:053
811822992018,1cyclictest0-21swapper/319:35:253
811822992017,2cyclictest910244-21seq23:45:423
811822992014,4cyclictest42-21ksoftirqd/322:10:013
81182299200,8cyclictest890250-21/usr/sbin/munin22:50:193
81182299200,1cyclictest3056-21PK-Backend19:17:273
81182299200,19cyclictest0-21swapper/321:23:473
81182299200,19cyclictest0-21swapper/321:13:053
81182299200,19cyclictest0-21swapper/321:07:523
81182299200,19cyclictest0-21swapper/321:01:023
81182299200,19cyclictest0-21swapper/320:29:123
81182299200,19cyclictest0-21swapper/300:32:533
811818992018,1cyclictest3056-21PK-Backend20:35:052
811818992018,1cyclictest0-21swapper/223:58:102
81181899201,7cyclictest835017-21apt20:15:012
81181899200,7cyclictest816716-21awk19:20:172
81181899200,6cyclictest912343-21awk23:55:002
81181899200,4cyclictest812739-21ntp_states19:10:202
81181899200,19cyclictest0-21swapper/222:33:482
81181599202,13cyclictest812866-21runrttasks19:10:221
811815992018,1cyclictest3056-21PK-Backend23:45:171
811815992018,1cyclictest3056-21PK-Backend00:15:191
811815992016,3cyclictest1095-21runrttasks20:13:341
811815992012,6cyclictest842955-21ntpq20:35:181
811811992019,1cyclictest0-21swapper/020:59:020
811811992018,1cyclictest3056-21PK-Backend21:10:170
811811992018,1cyclictest0-21swapper/023:07:580
811811992018,1cyclictest0-21swapper/022:53:190
811811992018,1cyclictest0-21swapper/022:38:450
811811992018,1cyclictest0-21swapper/022:22:230
811811992018,1cyclictest0-21swapper/021:27:210
811811992018,1cyclictest0-21swapper/020:39:350
811811992018,1cyclictest0-21swapper/020:02:040
811811992018,1cyclictest0-21swapper/019:42:030
81181199201,7cyclictest831828-21apt20:05:130
811811992017,2cyclictest0-21swapper/023:10:220
81181199200,1cyclictest3056-21PK-Backend22:00:060
81181199200,1cyclictest0-21swapper/022:49:340
81181199200,1cyclictest0-21swapper/000:39:390
81181199200,19cyclictest0-21swapper/022:07:220
81181199200,19cyclictest0-21swapper/000:15:390
81181199200,19cyclictest0-21swapper/000:00:240
811822991917,1cyclictest3056-21PK-Backend20:02:083
811822991916,2cyclictest3056-21PK-Backend20:55:153
81182299190,6cyclictest921630-21expr00:20:143
81182299190,1cyclictest3056-21PK-Backend20:43:243
81182299190,1cyclictest3056-21PK-Backend20:10:183
81182299190,1cyclictest3056-21PK-Backend19:42:073
81182299190,1cyclictest3056-21PK-Backend19:22:003
81182299190,1cyclictest0-21swapper/323:28:593
81182299190,1cyclictest0-21swapper/323:28:593
81182299190,1cyclictest0-21swapper/323:09:193
81182299190,1cyclictest0-21swapper/321:53:553
81182299190,1cyclictest0-21swapper/321:49:563
81182299190,18cyclictest0-21swapper/320:32:373
81181899191,3cyclictest978-21snmpd20:09:112
811815991915,3cyclictest829964-21date20:00:011
811815991911,6cyclictest902341-21grep23:25:181
811815991911,6cyclictest902341-21grep23:25:171
811811991918,1cyclictest0-21swapper/019:55:550
811811991917,1cyclictest0-21swapper/023:45:100
811811991917,1cyclictest0-21swapper/023:01:030
811811991917,1cyclictest0-21swapper/022:17:500
811811991915,3cyclictest866745-21sh21:45:000
81181199190,1cyclictest0-21swapper/021:59:180
81181199190,1cyclictest0-21swapper/020:11:300
81181199190,1cyclictest0-21swapper/019:31:150
81181199190,1cyclictest0-21swapper/000:29:450
81181199190,1cyclictest0-21swapper/000:29:440
81181199190,1cyclictest0-21swapper/000:24:170
81181199190,18cyclictest0-21swapper/023:27:400
81181199190,18cyclictest0-21swapper/023:27:400
811822991816,1cyclictest3056-21PK-Backend23:57:323
811822991816,1cyclictest3056-21PK-Backend19:11:463
811822991816,1cyclictest0-21swapper/320:19:123
811822991815,2cyclictest3056-21PK-Backend22:10:303
811822991811,5cyclictest910838-21/usr/sbin/munin23:50:193
81182299180,16cyclictest891894-21missed_timers22:55:183
81181599189,7cyclictest900686-21ntp_states23:20:191
811815991815,2cyclictest3056-21PK-Backend00:37:221
811815991813,3cyclictest841965-21sh20:34:591
811815991812,4cyclictest2156-21gnome-shell23:58:091
811815991811,5cyclictest823451-21cat19:40:171
811815991810,6cyclictest877804-21awk22:15:171
811811991816,1cyclictest0-21swapper/019:53:520
811822991715,1cyclictest0-21swapper/321:40:373
81182299171,14cyclictest923387-21df_inode00:25:143
81182299171,14cyclictest923387-21df_inode00:25:143
811822991710,5cyclictest881308-21cat22:25:173
81181599179,6cyclictest858068-21acpi21:20:111
81181599174,3cyclictest141rcu_preempt19:29:591
81181599173,11cyclictest831544-21cron20:05:001
81181599172,13cyclictest884499-21cut22:35:141
81181599172,13cyclictest828090-21munin-run19:55:001
811815991715,1cyclictest3056-21PK-Backend21:25:381
811815991715,1cyclictest3056-21PK-Backend20:15:491
811815991715,1cyclictest3056-21PK-Backend19:38:381
811815991714,2cyclictest3056-21PK-Backend00:02:341
811815991713,3cyclictest890453-21switchtime22:50:221
811815991710,5cyclictest876536-21python322:10:181
811815991710,4cyclictest873724-21latency_hist22:05:011
811811991711,5cyclictest899744-21sed23:20:000
811822991613,2cyclictest3056-21PK-Backend22:33:023
81182299160,14cyclictest837770-21ntp_states20:20:193
81181599168,6cyclictest908599-21apt-get23:45:001
81181599168,6cyclictest845449-21dpkg20:45:001
81181599168,5cyclictest816838-21switchtime19:20:191
81181599162,12cyclictest832489-21head20:05:191
811815991614,1cyclictest3056-21PK-Backend23:05:501
811815991614,1cyclictest0-21swapper/121:01:521
811815991611,4cyclictest26-21ksoftirqd/122:29:591
811815991611,3cyclictest926202-21cron00:35:001
811815991610,5cyclictest854755-21tr21:10:121
811815991610,5cyclictest846596-21sed20:45:171
81181599160,15cyclictest3056-21PK-Backend21:39:111
811811991614,1cyclictest0-21swapper/020:54:500
811815991513,1cyclictest3056-21PK-Backend22:40:551
811815991513,1cyclictest3056-21PK-Backend21:46:341
811815991513,1cyclictest3056-21PK-Backend21:16:261
811815991513,1cyclictest3056-21PK-Backend20:25:361
811815991513,1cyclictest3056-21PK-Backend20:22:051
81181599150,14cyclictest3056-21PK-Backend00:29:521
81181599150,14cyclictest3056-21PK-Backend00:29:511
81181599150,14cyclictest3056-21PK-Backend00:14:311
81181599150,14cyclictest3056-21PK-Backend00:06:061
811815991410,3cyclictest3056-21PK-Backend21:52:111
81181599140,3cyclictest3056-21PK-Backend20:50:191
81181599140,1cyclictest3056-21PK-Backend23:19:011
81181599140,13cyclictest3056-21PK-Backend23:11:041
81181599140,13cyclictest3056-21PK-Backend21:33:571
81181599140,13cyclictest3056-21PK-Backend21:09:431
81181599139,3cyclictest3056-21PK-Backend23:01:511
811815991311,1cyclictest3056-21PK-Backend22:24:531
81181599130,1cyclictest3056-21PK-Backend23:31:521
81181599100,8cyclictest825729-21chrt19:45:361
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional