You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-08 - 01:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Tue Jul 07, 2026 12:45:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
189132210170,34sleep20-21swapper/207:07:272
18793217398,26sleep10-21swapper/107:05:541
190902159109,27sleep30-21swapper/307:09:453
189662158108,27sleep00-21swapper/007:08:100
220121430,2sleep10-21swapper/109:30:141
374421350,1sleep10-21swapper/111:31:381
73362670,2sleep30-21swapper/310:21:493
309972560,2sleep10-21swapper/112:15:451
239452560,2sleep00-21swapper/011:46:420
164962560,2sleep20-21swapper/211:40:582
308692550,3sleep3560-21dbus-daemon08:50:003
300952500,2sleep00-21swapper/009:50:240
24022500,1sleep32401-21ssh12:19:363
92912440,2sleep10-21swapper/111:35:271
189662440,1sleep218968-21sh09:42:242
147442440,1sleep20-21swapper/209:39:552
174012430,1sleep00-21swapper/011:41:530
199322400,1sleep10-21swapper/110:55:211
290622370,2chrt0-21swapper/312:39:263
201322330,10sleep012-21ksoftirqd/009:43:340
19382993018,8cyclictest14583-21python311:15:240
19383992923,4cyclictest22115-21diskstats07:15:161
279952280,2chrt0-21swapper/208:40:242
1938299281,25cyclictest0-21swapper/007:18:480
19385992720,5cyclictest23565-21sshd07:18:553
19382992721,4cyclictest12-21ksoftirqd/011:00:010
226652260,2sleep30-21swapper/310:57:333
1938499262,16cyclictest6568-21ssh11:10:122
19384992612,9cyclictest52950irq/141-enp3s0-12:20:142
92012250,3sleep10-21swapper/112:24:401
1938499251,23cyclictest0-21swapper/207:15:252
19382992523,1cyclictest0-21swapper/007:21:080
19382992523,1cyclictest0-21swapper/007:21:070
1938299252,16cyclictest7017-21diskmemload12:15:140
84172240,2sleep10-21swapper/109:59:121
19385992412,10cyclictest29988-21sendmail_mailtr09:50:233
1938599241,20cyclictest737-21diskstats07:40:143
19385992411,11cyclictest23849-21expr10:10:163
19384992422,1cyclictest0-21swapper/207:22:042
19384992422,1cyclictest0-21swapper/207:22:032
1938499242,16cyclictest32285-21ssh11:05:132
1938499242,15cyclictest5384-21ssh10:45:122
19383992422,1cyclictest0-21swapper/107:20:021
19383992422,1cyclictest0-21swapper/107:20:011
19383992413,8cyclictest30327-21df_inode10:15:141
19383992410,8cyclictest52850irq/140-enp3s0-12:30:121
69952230,2sleep00-21swapper/011:34:540
1938599231,18cyclictest13265-21ssh11:15:133
1938499239,10cyclictest648-21ntpd10:00:212
19383992321,1cyclictest0-21swapper/107:36:251
1938399231,9cyclictest232602sh11:46:001
19382992311,6cyclictest8747-21cat09:35:130
1938499221,5cyclictest0-21swapper/212:39:102
19383992212,7cyclictest15496-21python309:15:251
19383992211,8cyclictest11863-21df08:05:141
19382992215,4cyclictest32552-21sh07:40:010
19382992211,7cyclictest9973-21apt-get12:25:120
298292210,2sleep00-21swapper/007:30:270
19385992115,4cyclictest27126-21sh09:25:013
19385992113,6cyclictest22458-21expr12:10:113
19385992113,6cyclictest16611-21sensors_temp09:40:233
19385992112,7cyclictest26994-21df11:25:143
19385992112,7cyclictest26994-21df11:25:143
19385992112,6cyclictest32175-21seq10:40:213
19385992112,6cyclictest12005-21fw_packets08:05:163
19384992119,1cyclictest0-21swapper/207:29:432
19384992113,6cyclictest22904-21cut08:30:172
19384992112,6cyclictest13415-21cron08:09:592
19383992114,4cyclictest18597-21sh11:43:051
19383992113,6cyclictest25773-21which08:35:231
19383992113,5cyclictest31840-21sh11:05:011
19383992112,7cyclictest9856-21seq10:00:121
19383992112,6cyclictest5453-21df09:05:141
19383992111,7cyclictest19896-21lsb_release08:22:081
19383992111,7cyclictest10621-21cat08:00:231
1938299212,13cyclictest636-21runrttasks11:25:120
1938299212,13cyclictest636-21runrttasks11:25:120
19382992115,4cyclictest12-21ksoftirqd/012:40:000
19382992112,7cyclictest19397-21awk10:30:250
19382992112,6cyclictest867-21fw_packets07:40:150
19382992111,7cyclictest23871-21sed08:30:270
19382992110,7cyclictest5130-21apt-get09:05:130
315142201,4sleep331669-21hddtemp_smartct08:50:173
313472200,3sleep10-21swapper/111:52:131
1938599209,8cyclictest7877-21df_inode11:35:133
1938599202,14cyclictest0-21swapper/309:55:113
19385992014,4cyclictest10189-21sh11:12:223
19385992013,5cyclictest23728-21uptime09:45:253
19385992013,4cyclictest8911-21timerandwakeup09:10:233
19385992012,6cyclictest8393-21sensors_temp07:55:233
19385992012,6cyclictest6081-21pluginstate09:05:223
19385992012,6cyclictest20741-21entropy08:25:163
19385992012,6cyclictest1747-21cat08:55:193
19385992011,7cyclictest19973-21apt-get11:20:113
19385992011,6cyclictest10964-21cpu10:25:123
19385992010,6cyclictest53050irq/142-enp3s0-09:30:123
1938499209,9cyclictest31057-21/usr/sbin/munin10:40:132
1938499209,7cyclictest10956-21netstat12:25:192
1938499201,8cyclictest2926-21apt-get09:55:122
1938499201,5cyclictest0-21swapper/210:31:262
19384992013,5cyclictest7650-21expr09:10:132
19384992012,6cyclictest18513-21cut10:55:002
19384992012,5cyclictest636-21runrttasks11:02:392
19384992011,6cyclictest560-21dbus-daemon07:50:002
19384992011,6cyclictest3349-21sed09:30:252
19384992011,6cyclictest29735-21memory08:45:192
19384992011,6cyclictest27546-21date11:50:012
19384992011,6cyclictest20417-21cron09:20:002
1938399209,7cyclictest52850irq/140-enp3s0-09:35:131
1938399208,8cyclictest31087-21apt-get10:40:111
1938399202,12cyclictest28977-21apt-get08:45:141
1938399202,12cyclictest17695-21apt-get10:30:141
1938399202,12cyclictest15815-21apt-get08:15:101
19383992014,4cyclictest25656-21sh09:23:311
19383992013,5cyclictest14877-21seq10:51:281
19383992012,6cyclictest4806-21seq11:08:411
19383992012,6cyclictest15900-21cat12:05:131
1938399201,17cyclictest636-21runrttasks11:21:231
19383992011,7cyclictest23744-21entropy12:35:141
19383992011,6cyclictest5008-21cat09:05:011
19383992011,6cyclictest5008-21cat09:05:011
1938399201,15cyclictest23313-21ssh10:10:131
1938299202,15cyclictest3077-21tr09:55:120
1938299202,14cyclictest21971-21grep09:20:220
1938299202,12cyclictest24679-21apt-get08:35:130
1938299202,12cyclictest2127-21apt-get11:55:130
1938299202,12cyclictest16577-21ssh10:05:130
19382992014,3cyclictest9868-21/usr/sbin/munin10:00:120
19382992012,6cyclictest9090-21grep09:10:250
19382992012,6cyclictest8215-21cut09:35:000
19382992012,5cyclictest8424-21sed11:35:180
19382992011,7cyclictest18734-21seq12:06:530
19382992011,6cyclictest18422-21grep12:30:270
19382992011,6cyclictest16336-21sed08:15:140
107872200,3sleep326755-1kworker/u9:112:00:393
278342190,3sleep27017-21diskmemload09:25:142
243672192,4sleep122-21ksoftirqd/109:46:051
1938599192,12cyclictest5374-21apt-get10:45:133
1938599192,11cyclictest2757-21apt-get07:45:133
1938599191,6cyclictest0-21swapper/311:47:483
19385991913,4cyclictest26636-21sh10:35:573
19385991913,4cyclictest26636-21sh10:35:573
19385991913,4cyclictest22887-21sh10:33:543
19385991913,4cyclictest16689-21sh12:30:113
19385991912,5cyclictest53050irq/142-enp3s0-12:20:023
19385991912,5cyclictest4797-21seq09:04:093
19385991912,5cyclictest4797-21seq09:04:083
19385991912,5cyclictest30638-21cat07:35:143
19385991912,5cyclictest29531-21ssh11:02:423
19385991912,5cyclictest14215-21awk11:40:113
19385991912,4cyclictest13968-21sh09:15:033
19385991911,6cyclictest6274-21cat11:10:003
19385991911,6cyclictest4206-21seq11:32:053
19385991911,6cyclictest30311-21cat10:15:143
1938599191,16cyclictest28407-21gltestperf11:50:163
19385991911,6cyclictest14983-21cut09:39:593
19385991911,6cyclictest12772-21if_enp3s010:50:163
19385991911,6cyclictest11412-21cat08:05:003
19385991911,6cyclictest11312-21python312:25:233
1938599191,15cyclictest2306-21tr11:55:133
1938499199,3cyclictest26641-21sh10:35:582
1938499199,3cyclictest26641-21sh10:35:572
19384991912,4cyclictest1361-21sh10:17:332
19384991911,6cyclictest9357-21sed08:00:002
19384991911,6cyclictest278-21systemd-journal12:35:012
19384991910,6cyclictest311-21sed07:40:012
1938399193,10cyclictest4246-21ssh10:20:131
19383991912,5cyclictest796-21seq08:55:071
19383991912,5cyclictest16966-21forks10:05:141
19383991911,6cyclictest7240-21apt-get07:55:111
19383991911,6cyclictest6417-21tr10:45:201
19383991911,6cyclictest23187-21needreboot12:10:191
19383991911,5cyclictest52850irq/140-enp3s0-09:25:251
1938299199,7cyclictest3699-21cat09:00:170
1938299199,7cyclictest3699-21cat09:00:160
1938299199,6cyclictest29953-21apt-get10:15:120
1938299198,8cyclictest26402-21ssh11:00:180
1938299194,5cyclictest636-21runrttasks10:27:500
1938299192,10cyclictest22016-21ssh09:45:110
1938299192,10cyclictest1393-21gpm08:40:140
1938299191,6cyclictest29928-21taskset10:39:160
1938299191,6cyclictest29928-21taskset10:39:160
19382991914,3cyclictest636-21runrttasks10:11:540
19382991913,4cyclictest12-21ksoftirqd/011:05:230
19382991911,6cyclictest9830-21tune2fs08:00:160
19382991911,6cyclictest31853-21sed08:50:190
150622190,2sleep10-21swapper/110:27:491
1938599187,9cyclictest0-21swapper/310:08:133
1938599182,11cyclictest4892-21apt-get07:50:103
19385991813,3cyclictest27967-21latency_hist07:30:013
19385991813,3cyclictest24491-21hddtemp_smartct07:20:163
19385991813,3cyclictest24491-21hddtemp_smartct07:20:163
19385991813,3cyclictest23461-21ntp_states08:30:233
19385991812,4cyclictest19698-21sh12:07:513
19385991812,4cyclictest16139-21sh08:15:123
19385991811,5cyclictest28018-21expr08:40:243
19385991811,4cyclictest0-21swapper/309:25:133
19385991810,6cyclictest9981-21cat10:00:143
19385991810,5cyclictest25808-21run-parts08:35:243
1938499189,7cyclictest17338-21cut10:05:182
1938499189,6cyclictest52950irq/141-enp3s0-11:25:242
1938499189,6cyclictest52950irq/141-enp3s0-11:25:232
1938499188,7cyclictest0-21swapper/209:00:002
1938499184,3cyclictest141rcu_preempt12:00:252
1938499183,5cyclictest636-21runrttasks11:37:552
1938499182,4cyclictest29736-21irqstats12:15:192
19384991812,4cyclictest8977-21sh10:23:292
19384991812,4cyclictest7017-21diskmemload09:47:342
19384991812,4cyclictest13985-21sh10:26:442
19384991812,3cyclictest21398-21proc_pri08:25:222
19384991811,5cyclictest6931-21apt-key07:55:002
19384991811,5cyclictest29265-21cat11:50:242
19384991811,4cyclictest7017-21diskmemload12:12:372
1938499181,14cyclictest28831-21tr09:50:122
19384991811,4cyclictest16938-21sh12:05:232
19384991811,4cyclictest0-21swapper/211:30:152
19384991810,6cyclictest9863-21fw_packets08:00:162
19384991810,6cyclictest3077-21needreboot11:55:202
19384991810,5cyclictest19285-21uniq10:55:162
1938399189,6cyclictest31456-21cat08:50:151
1938399187,7cyclictest25038-21hwlatdetect10:35:171
1938399187,7cyclictest25038-21hwlatdetect10:35:171
1938399182,11cyclictest15283-21apt-get09:40:131
19383991813,3cyclictest26483-21sh11:25:031
19383991813,3cyclictest26483-21sh11:25:031
19383991813,3cyclictest14005-21irqstats11:15:191
19383991813,3cyclictest10634-21hwlatdetect12:25:171
19383991812,4cyclictest4529-21sh11:56:061
19383991812,4cyclictest31626-21sh09:51:391
19383991812,4cyclictest22-21ksoftirqd/107:54:591
19383991812,4cyclictest11563-21sh09:12:511
19383991812,4cyclictest10752-21sh12:00:361
19383991811,5cyclictest29381-21pmu-power07:30:241
1938399181,12cyclictest363-21apt-get07:40:131
19383991810,6cyclictest22142-21seq08:28:221
1938299189,6cyclictest4355-21ntpq12:20:220
1938299189,6cyclictest20757-21sed07:10:240
1938299184,3cyclictest7017-21diskmemload11:11:550
1938299182,11cyclictest4227-21apt-get10:20:120
19382991813,3cyclictest0-21swapper/012:00:290
19382991812,4cyclictest29155-21cstates08:45:140
19382991811,5cyclictest14488-21expr09:15:140
19382991811,4cyclictest7017-21diskmemload11:23:280
1938299181,12cyclictest11542-21apt-get08:05:130
19382991810,6cyclictest12544-21cut10:50:150
19382991810,5cyclictest26507-21tune2fs07:25:140
1938599179,6cyclictest19729-21ls07:10:133
1938599178,6cyclictest19674-21gpgv08:22:033
1938599171,11cyclictest13679-21apt-get08:10:123
1938599171,11cyclictest13679-21apt-get08:10:113
1938499179,6cyclictest20304-21ntp_kernel_err07:10:202
1938499178,6cyclictest278-21systemd-journal07:30:012
1938499178,6cyclictest14186-21gltestperf08:10:162
1938499178,6cyclictest14186-21gltestperf08:10:162
1938499177,6cyclictest52950irq/141-enp3s0-11:15:122
1938499171,7cyclictest25572-21ssh10:11:002
19384991712,3cyclictest17957-21/usr/sbin/munin08:20:152
19384991712,3cyclictest1115-21latency07:40:192
19384991711,3cyclictest7017-21diskmemload09:23:132
1938499171,10cyclictest2991-21apt-get09:00:122
1938499171,10cyclictest2991-21apt-get09:00:122
19384991710,5cyclictest6425-21sed09:05:252
19384991710,5cyclictest19913-21seq11:20:102
1938399178,7cyclictest21162-21seq07:13:191
1938399172,9cyclictest26838-21apt-get08:40:121
19383991712,3cyclictest2732-21/usr/sbin/munin07:45:251
19383991712,3cyclictest22540-21apt08:30:121
1938399171,11cyclictest26074-21apt-get07:25:101
1938399170,15cyclictest6527-21cat11:10:111
1938299177,7cyclictest31823-21irqstats10:40:170
19382991713,2cyclictest18896-21ntp_states08:20:210
19382991712,3cyclictest636-21runrttasks11:50:430
19382991712,3cyclictest27519-21/usr/sbin/munin09:25:140
19382991712,3cyclictest23471-21processes12:10:210
1938299171,13cyclictest7537-21ssh10:45:510
19382991710,5cyclictest8309-21cut07:55:220
19382991710,5cyclictest13977-21expr08:10:130
19382991710,5cyclictest13977-21expr08:10:120
19382991710,4cyclictest812-21acpi08:55:100
19385991611,3cyclictest28405-21acpi07:30:133
1938499168,6cyclictest25315-21cat08:35:172
19384991611,3cyclictest31268-21cstates08:50:142
19384991611,3cyclictest15890-21cpuspeed_turbos08:15:112
1938399168,6cyclictest15267-21seq08:11:211
1938399168,6cyclictest15267-21seq08:11:201
1938299162,9cyclictest52750irq/139-enp3s0-08:25:120
19382991611,3cyclictest5301-21fschecks_time07:50:140
19382991610,4cyclictest2983-21expr07:45:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional