You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-24 - 23:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Mon Aug 24, 2026 12:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
13512996929,27cyclictest35966irq/58-eth010:00:153
1350499622,50cyclictest7352-21rm11:20:132
1348899564,42cyclictest8790-21ssh09:15:130
13488995520,24cyclictest12-21ksoftirqd/008:05:130
1348899534,39cyclictest19834-21ssh11:45:130
1350499523,36cyclictest28877-21ssh11:50:132
1349699522,31cyclictest10917-21ssh11:40:131
13504994812,26cyclictest0-21swapper/208:25:132
1350499472,9cyclictest131rcu_preempt11:05:142
1350499454,3cyclictest0-21swapper/209:05:132
1349699453,31cyclictest26937-21ssh09:25:131
1348899451,33cyclictest14598-21rm10:30:140
13504994418,17cyclictest0-21swapper/210:00:132
13504994219,20cyclictest27346irq/40-dwc2_hso09:00:132
1350499418,30cyclictest0-21swapper/207:35:132
13504994128,9cyclictest17448-21apt-get08:55:132
1348899416,22cyclictest32677-21ssh10:40:150
1348899416,22cyclictest32677-21ssh10:40:150
1350499406,12cyclictest6947-21munin-run11:20:012
1350499402,30cyclictest10969-21ssh10:10:132
1350499402,30cyclictest10969-21ssh10:10:122
13504994017,20cyclictest23548-21ssh10:35:112
1349699406,25cyclictest25357-21ssh10:00:131
1349699391,30cyclictest30675-21sh09:45:141
1349699382,29cyclictest10144-21sshd07:40:101
1348899389,24cyclictest12-21ksoftirqd/007:50:120
1348899383,4cyclictest131rcu_preempt12:35:140
13488993810,20cyclictest12-21ksoftirqd/009:20:110
1350499374,30cyclictest28771-21kworker/u8:212:20:122
13504993732,3cyclictest27-21ksoftirqd/210:50:112
13504993720,12cyclictest21692-21apt-get09:40:142
13496993724,8cyclictest31639-21diskmemload10:15:121
13496993724,8cyclictest31639-21diskmemload10:15:121
1348899371,23cyclictest0-21swapper/008:35:130
1351299363,20cyclictest35966irq/58-eth011:20:133
1350499365,22cyclictest14263-21ssh10:45:492
1351299355,17cyclictest35966irq/58-eth011:50:123
1351299353,21cyclictest35966irq/58-eth012:15:143
1351299351,22cyclictest35966irq/58-eth010:45:133
1350499356,11cyclictest0-21swapper/209:10:432
1350499354,15cyclictest12130-21ssh10:27:332
13504993517,13cyclictest5496-21ssh11:55:132
1350499351,29cyclictest14554-21ssh12:00:142
1350499351,29cyclictest14554-21ssh12:00:142
1348899356,24cyclictest12-21ksoftirqd/012:20:120
1348899352,23cyclictest12-21ksoftirqd/011:10:150
13488993519,10cyclictest32558-21sh12:10:140
1350499347,17cyclictest31639-21diskmemload11:00:372
1350499346,14cyclictest26210-21irqcore08:30:172
13504993424,4cyclictest1897-21apt-get11:35:152
13504993423,3cyclictest3526-21ssh09:30:122
1348899342,25cyclictest7297-21ssh09:50:150
1351299333,16cyclictest35966irq/58-eth009:25:123
1350499336,19cyclictest20385-21ssh09:38:572
1350499335,18cyclictest0-21swapper/209:47:392
1350499335,18cyclictest0-21swapper/209:18:482
13496993313,5cyclictest131rcu_preempt12:35:131
1348899331,24cyclictest21695-21ssh09:40:130
1348899326,19cyclictest3965-21ssh12:30:150
1348899324,21cyclictest25283-21ssh11:30:140
1348899321,21cyclictest19955-21sh10:15:110
1348899321,21cyclictest19955-21sh10:15:100
1350499315,17cyclictest27346irq/40-dwc2_hso11:40:152
13504993119,8cyclictest12955-21apt-get12:35:152
13504993116,10cyclictest23583-21ssh12:05:132
13504993115,13cyclictest26689-21sshd11:10:382
1348899314,16cyclictest28993-21ssh10:20:130
1351299303,16cyclictest35966irq/58-eth009:30:133
1350499304,11cyclictest0-21swapper/210:19:192
1350499304,11cyclictest0-21swapper/210:19:192
13504993023,3cyclictest25251-21apt-get11:30:142
13504993019,3cyclictest12636-21apt-get08:50:152
13504993012,14cyclictest0-21swapper/210:20:172
13504993011,10cyclictest7562-21cut08:10:202
13504993010,11cyclictest18089-21cut09:20:152
1349699302,18cyclictest6091-1kworker/u9:210:45:131
1348899303,20cyclictest1098466ssh10:10:140
1348899303,20cyclictest1098466ssh10:10:130
13488993013,10cyclictest12-21ksoftirqd/010:50:110
1351299292,14cyclictest35966irq/58-eth009:40:133
1350499298,18cyclictest5403-21/usr/sbin/munin08:05:282
13504992913,13cyclictest19857-21fschecks_count07:50:142
1350499289,14cyclictest0-21swapper/212:15:132
1350499283,17cyclictest27346irq/40-dwc2_hso12:25:112
13504992821,5cyclictest27346irq/40-dwc2_hso12:30:242
13504992811,14cyclictest10538-21apt-config10:10:022
13504992811,13cyclictest32671-21ssh10:40:142
13504992811,13cyclictest32671-21ssh10:40:132
13504992810,15cyclictest16355-21grep09:55:152
1349699282,20cyclictest12798-21ssh11:05:151
13488992819,2cyclictest0-21swapper/011:00:140
1350499275,8cyclictest0-21swapper/209:27:452
1350499274,17cyclictest27346irq/40-dwc2_hso11:25:142
1350499273,17cyclictest27346irq/40-dwc2_hso07:30:112
1350499272,19cyclictest0-21swapper/207:45:132
1350499272,16cyclictest27346irq/40-dwc2_hso08:00:142
13504992716,8cyclictest11373-21apt-get08:15:142
13504992711,13cyclictest3500-21sed11:00:022
13504992710,13cyclictest14358-21kthreadcore07:10:192
1350499269,14cyclictest18473-21cpuspeed07:15:142
1350499262,16cyclictest27346irq/40-dwc2_hso08:40:132
1350499262,11cyclictest0-21swapper/212:11:552
13504992612,11cyclictest10915-21ntp_192.168.11409:50:252
13504992612,10cyclictest0-21swapper/207:20:132
13504992610,14cyclictest21138-21ssh10:32:352
13504992610,13cyclictest30600-21find08:35:122
1349699263,18cyclictest9149-21ssh12:15:131
1349699263,13cyclictest17452-21sshd08:55:131
13496992614,9cyclictest21-21ksoftirqd/111:00:291
13496992612,10cyclictest2755-21cron08:40:011
1348899263,18cyclictest5515-21ssh11:55:150
1348899262,17cyclictest23555-21rm10:35:130
1351299254,12cyclictest35966irq/58-eth010:30:123
1350499259,13cyclictest11607-21sort08:45:302
1350499259,13cyclictest11607-21sort08:45:302
1349699254,4cyclictest201rcuc/110:35:131
13496992511,7cyclictest0-21swapper/109:35:121
1348899252,15cyclictest12769-21ssh11:05:130
13512992416,5cyclictest321rcuc/311:45:123
1350499249,12cyclictest24516-21/usr/sbin/munin07:55:112
1350499242,13cyclictest23538-21ntp_states11:45:252
1349699248,5cyclictest131rcu_preempt11:45:131
1348899246,13cyclictest14570-21diskmemload12:00:150
1348899246,13cyclictest14570-21diskmemload12:00:150
1348899241,19cyclictest5621-21ssh11:35:270
1348899241,17cyclictest30682-21ssh09:45:150
1351299233,16cyclictest35966irq/58-eth008:35:133
1351299232,17cyclictest0-21swapper/310:10:133
1351299232,17cyclictest0-21swapper/310:10:123
13512992317,4cyclictest321rcuc/310:07:273
13512992312,4cyclictest0-21swapper/311:40:133
1350499235,6cyclictest0-21swapper/207:40:022
1350499235,10cyclictest930-21snmpd07:25:312
1350499232,15cyclictest27346irq/40-dwc2_hso08:20:132
1350499232,15cyclictest27346irq/40-dwc2_hso08:20:132
1349699239,4cyclictest21-21ksoftirqd/107:45:181
1349699235,13cyclictest27340-21ssh12:25:131
13496992315,6cyclictest2018-21ssh10:05:151
13496992314,6cyclictest18128-21ssh12:20:041
1349699231,17cyclictest25380-21ssh11:10:251
1349699231,14cyclictest28882-21ssh11:50:131
13496992310,4cyclictest0-21swapper/112:09:191
1348899233,15cyclictest24953-21munin-run10:00:020
1351299227,11cyclictest35966irq/58-eth009:35:093
1351299222,11cyclictest0-21swapper/311:35:043
13512992217,3cyclictest321rcuc/309:13:103
13512992217,3cyclictest321rcuc/308:25:133
13512992215,5cyclictest321rcuc/308:05:143
13512992213,6cyclictest35966irq/58-eth009:55:133
1351299221,13cyclictest35966irq/58-eth010:20:133
1351299220,13cyclictest35966irq/58-eth012:35:153
13496992215,5cyclictest0-21swapper/110:55:121
13496992214,6cyclictest2231-21ssh11:35:181
13496992213,6cyclictest30260-21cron09:45:011
1349699221,18cyclictest16942-21ls07:10:231
13496992210,4cyclictest21-21ksoftirqd/110:20:161
13496992210,10cyclictest21-21ksoftirqd/107:22:371
1348899224,15cyclictest12-21ksoftirqd/010:25:130
13488992213,6cyclictest997-21sshd10:57:100
1348899221,17cyclictest1930-21sh10:05:130
1348899221,16cyclictest6722-21cut08:10:170
1351299215,11cyclictest35966irq/58-eth012:20:113
1351299214,10cyclictest35966irq/58-eth012:10:123
1351299212,11cyclictest0-21swapper/312:05:143
13512992115,4cyclictest321rcuc/311:17:033
13512992115,4cyclictest321rcuc/311:10:143
13512992115,4cyclictest321rcuc/309:54:103
13512992113,6cyclictest321rcuc/309:45:063
1350499211,10cyclictest13578-21latency_hist07:10:022
1349699218,7cyclictest21-21ksoftirqd/107:25:181
1349699218,3cyclictest131rcu_preempt12:30:141
1349699217,4cyclictest21-21ksoftirqd/111:55:171
1349699215,4cyclictest131rcu_preempt10:50:121
1349699212,16cyclictest20199-21switchtime09:55:281
1349699211,3cyclictest0-21swapper/111:16:041
13496992110,9cyclictest26547-21cron09:25:021
13496992110,3cyclictest21-21ksoftirqd/111:21:311
13496992110,2cyclictest21-21ksoftirqd/109:19:081
1348899211,7cyclictest13484-21cut11:40:230
13488992111,7cyclictest30054-21cat10:00:330
13488992110,8cyclictest12-21ksoftirqd/010:45:140
1351299209,7cyclictest35966irq/58-eth009:20:313
1351299208,7cyclictest321rcuc/307:45:143
13512992015,3cyclictest321rcuc/309:00:263
1351299201,16cyclictest35966irq/58-eth012:30:133
1351299201,14cyclictest35966irq/58-eth012:25:133
1351299201,13cyclictest0-21swapper/307:25:233
13512992010,7cyclictest0-21swapper/311:25:333
1349699209,3cyclictest21-21ksoftirqd/112:00:151
1349699209,3cyclictest21-21ksoftirqd/112:00:141
1349699209,3cyclictest21-21ksoftirqd/110:25:151
1349699209,2cyclictest21-21ksoftirqd/109:31:281
1349699208,7cyclictest0-21swapper/111:34:221
1349699201,6cyclictest10048-21awk08:10:261
13496992014,4cyclictest0-21swapper/110:30:191
13496992013,5cyclictest0-21swapper/111:25:121
13496992012,5cyclictest0-21swapper/107:55:231
13496992011,7cyclictest0-21swapper/107:10:011
1348899207,10cyclictest614-21dbus-daemon07:44:530
1348899201,16cyclictest6728-21timerandwakeup08:40:280
1348899201,13cyclictest1343-21ssh11:50:340
1348899201,11cyclictest22979-21latency_hist07:20:020
1351299195,9cyclictest0-21swapper/308:20:133
1351299195,9cyclictest0-21swapper/308:20:123
1351299193,10cyclictest35966irq/58-eth007:55:143
1351299192,11cyclictest35966irq/58-eth011:55:153
13512991914,3cyclictest321rcuc/308:10:263
13512991913,4cyclictest321rcuc/310:25:153
13512991913,4cyclictest0-21swapper/311:30:343
13512991912,4cyclictest321rcuc/311:05:433
1351299191,10cyclictest0-21swapper/310:35:133
1349699199,7cyclictest5375-21df_inode07:35:151
1349699199,3cyclictest21-21ksoftirqd/110:40:181
1349699199,3cyclictest21-21ksoftirqd/110:40:171
1349699192,6cyclictest21-21ksoftirqd/109:10:011
13496991913,4cyclictest0-21swapper/112:10:151
1349699191,13cyclictest7287-21ssh09:50:141
1349699191,10cyclictest32089-21/usr/sbin/munin09:10:251
1349699190,16cyclictest15904-21ls08:50:251
1348899196,9cyclictest0-21swapper/011:15:130
1348899194,9cyclictest12-21ksoftirqd/008:15:140
1348899192,14cyclictest23438-21grep07:20:170
1348899191,9cyclictest17885-21kthreadcore11:25:200
13488991912,4cyclictest23542-21/usr/sbin/munin12:05:280
1348899191,15cyclictest12152-21apt-get09:35:050
13488991910,6cyclictest15921-21cron08:20:020
13488991910,6cyclictest15921-21cron08:20:010
1351299182,10cyclictest0-21swapper/307:30:123
13512991813,3cyclictest0-21swapper/310:40:203
13512991813,3cyclictest0-21swapper/310:40:203
13512991812,4cyclictest0-21swapper/310:55:433
13512991812,4cyclictest0-21swapper/310:53:473
13512991812,4cyclictest0-21swapper/310:17:343
13512991812,4cyclictest0-21swapper/310:17:343
13512991812,4cyclictest0-21swapper/309:16:143
13512991811,4cyclictest35966irq/58-eth011:00:133
1351299181,11cyclictest0-21swapper/308:35:023
1349699182,9cyclictest21-21ksoftirqd/108:20:181
1349699182,9cyclictest21-21ksoftirqd/108:20:171
13496991813,3cyclictest21-21ksoftirqd/108:40:201
13496991813,3cyclictest0-21swapper/109:00:281
13496991812,4cyclictest0-21swapper/110:10:231
13496991812,4cyclictest0-21swapper/110:10:231
13496991810,3cyclictest21-21ksoftirqd/107:15:181
1348899185,10cyclictest30074-21kthreadcore07:25:200
1348899185,10cyclictest0-21swapper/011:20:140
1348899182,8cyclictest12148-21munin-run09:35:020
1348899182,3cyclictest131rcu_preempt12:25:130
1348899182,13cyclictest9281-21diskstats12:15:160
1348899182,12cyclictest30405-21memory09:25:230
1348899181,14cyclictest8930-21cat07:35:270
1348899181,14cyclictest8486-21latency_hist09:15:010
1348899181,14cyclictest13564-21latency_hist07:10:020
1351299179,6cyclictest0-21swapper/312:00:123
1351299179,6cyclictest0-21swapper/312:00:113
1351299178,4cyclictest321rcuc/308:00:143
1351299175,7cyclictest0-21swapper/307:35:133
1351299172,10cyclictest0-21swapper/308:45:153
1351299172,10cyclictest0-21swapper/308:45:143
1351299172,10cyclictest0-21swapper/307:20:153
1351299172,10cyclictest0-21swapper/307:10:143
13512991711,4cyclictest0-21swapper/308:55:143
13512991711,4cyclictest0-21swapper/308:15:143
1351299171,10cyclictest0-21swapper/307:15:123
1349699178,7cyclictest14776-21ntp_kernel_err08:15:251
1349699178,2cyclictest21-21ksoftirqd/107:30:161
1349699172,10cyclictest0-21swapper/108:30:141
1349699171,8cyclictest21-21ksoftirqd/108:00:161
13496991712,3cyclictest0-21swapper/108:25:141
13496991711,4cyclictest0-21swapper/107:50:151
13496991710,4cyclictest5377-21grep08:05:281
1348899179,6cyclictest31000-21sshd09:05:290
1348899174,10cyclictest0-21swapper/008:50:150
1348899171,12cyclictest1-21systemd08:55:140
1348899171,11cyclictest1-21systemd08:00:010
1348899170,8cyclictest1428-21kthreadcore07:30:180
1348899170,14cyclictest15304-21cut07:45:170
1351299162,9cyclictest0-21swapper/307:40:123
13512991611,3cyclictest0-21swapper/308:50:193
13496991611,3cyclictest0-21swapper/108:45:171
13496991611,3cyclictest0-21swapper/108:45:161
1348899165,7cyclictest614-21dbus-daemon08:30:090
1348899164,9cyclictest15628-21kthreadcore07:10:200
1348899164,7cyclictest0-21swapper/008:45:140
1348899164,7cyclictest0-21swapper/008:45:130
1348899162,9cyclictest0-21swapper/009:00:130
1348899161,12cyclictest21553-21irqrtprio08:25:190
1348899161,10cyclictest32695-21ls08:00:240
13512991510,3cyclictest321rcuc/308:40:143
13512991510,3cyclictest0-21swapper/307:50:343
1351299148,4cyclictest0-21swapper/309:05:143
1351299101,6cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional